Oct 30 23:15:50.904649 ip-10-0-134-83 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 23:15:50.904661 ip-10-0-134-83 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 23:15:50.904670 ip-10-0-134-83 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 23:15:50.905000 ip-10-0-134-83 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 23:16:01.144282 ip-10-0-134-83 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 23:16:01.144298 ip-10-0-134-83 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ee1ef1d60d504082a5fccdfa7085208d --
Oct 30 23:18:19.774735 ip-10-0-134-83 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 23:18:20.189241 ip-10-0-134-83 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:18:20.189241 ip-10-0-134-83 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 23:18:20.189241 ip-10-0-134-83 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:18:20.189241 ip-10-0-134-83 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:18:20.192573 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192484    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:18:20.192573 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192572    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:18:20.192573 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192576    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192579    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192582    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192585    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192588    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192591    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192593    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192597    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192599    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192602    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192605    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192607    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192610    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192613    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192622    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192625    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192627    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192630    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192633    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:18:20.192668 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192635    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192638    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192641    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192644    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192647    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192649    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192652    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192655    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192657    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192660    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192662    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192665    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192667    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192670    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192673    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192676    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192679    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192682    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192685    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:18:20.193108 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192687    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192690    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192693    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192695    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192698    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192701    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192703    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192706    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192708    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192711    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192714    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192719    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192722    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192725    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192728    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192733    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192736    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192739    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192741    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:18:20.193562 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192744    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192747    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192749    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192752    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192755    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192758    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192760    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192763    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192766    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192768    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192771    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192773    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192776    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192779    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192782    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192784    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192787    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192789    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192792    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192794    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:18:20.193998 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192797    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192800    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192803    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192806    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192809    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192812    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192815    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192818    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192821    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192825    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192828    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192830    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192833    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192838    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192842    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192846    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192849    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192854    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192857    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:18:20.194515 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192860    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192863    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192866    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192869    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192872    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192875    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192877    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.192880    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193801    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193811    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193814    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193817    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193820    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193823    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193826    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193830    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193832    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193836    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193839    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193842    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:18:20.194965 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193845    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193847    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193850    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193853    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193855    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193858    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193861    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193864    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193866    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193869    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193872    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193874    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193877    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193880    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193884    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193889    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193892    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193896    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193898    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:18:20.195490 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193901    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193904    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193907    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193910    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193912    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193916    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193919    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193922    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193924    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193927    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193929    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193932    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193935    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193938    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193941    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193944    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193946    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193950    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193953    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193956    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:18:20.195937 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193958    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193961    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193964    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193967    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193969    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193972    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193974    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193977    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193979    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193982    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193985    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193987    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193989    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193993    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193996    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.193998    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194001    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194004    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194007    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194010    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:18:20.196446 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194012    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194015    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194018    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194020    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194023    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194026    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194028    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194031    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194033    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194036    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194039    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194041    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194044    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194047    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194049    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194052    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194054    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194057    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194059    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194062    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:18:20.196956 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194064    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194067    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194070    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194072    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194076    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194081    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194084    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194087    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194091    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194094    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194097    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194101    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194104    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194106    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194110    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194689    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194696    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194700    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194703    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:18:20.197473 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194706    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194709    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194711    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194714    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194717    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194720    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194723    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194725    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194728    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194731    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194734    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194736    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194739    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194742    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194745    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194748    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194751    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194754    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194757    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194760    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:18:20.197923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194763    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194765    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194768    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194771    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194773    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194776    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194779    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194781    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194784    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194787    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194790    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194792    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194795    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194798    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194800    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194803    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194805    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194808    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194811    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194813    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:18:20.198402 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194816    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194821    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194825    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194830    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194833    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194835    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194838    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194840    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194843    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194846    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194849    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194852    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194854    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194857    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194860    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194863    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194865    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194868    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194871    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194873    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:18:20.198876 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194876    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194879    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194882    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194885    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194888    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194890    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194893    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194896    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194898    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194901    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194903    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194906    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194908    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194912    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194915    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194917    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194920    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194922    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194925    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194928    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:18:20.199416 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194931    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194933    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194936    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194939    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194941    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194946    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194950    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194953    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194956    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194959    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194962    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194965    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194968    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194971    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194974    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194977    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194980    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194982    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194985    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:18:20.199895 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194988    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194991    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.194993    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195098    2571 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195106    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195112    2571 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195119    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195124    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195128    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195132    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195137    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195140    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195144    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195147    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195150    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195154    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195157    2571 flags.go:64] FLAG: --cgroup-root=""
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195160    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195164    2571 flags.go:64] FLAG: --client-ca-file=""
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195167    2571 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195170    2571 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195175    2571 flags.go:64] FLAG: --cluster-domain=""
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195178    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195182    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 23:18:20.200359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195185    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195189    2571 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195193    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195196    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195199    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195203    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195207    2571 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195210    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195213    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195217    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195220    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195224    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195228    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195231    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195236    2571 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195240    2571 flags.go:64] FLAG: --enable-server="true"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195244    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195248    2571 flags.go:64] FLAG: --event-burst="100"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195252    2571 flags.go:64] FLAG: --event-qps="50"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195255    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195258    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195261    2571 flags.go:64] FLAG: --eviction-hard=""
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195265    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195269    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195272    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 23:18:20.201036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195275    2571 flags.go:64] FLAG: --eviction-soft=""
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195279    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195282    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195285    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195288    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195291    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195294    2571 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195298    2571 flags.go:64] FLAG: --feature-gates=""
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195302    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195306    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195309    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195312    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195316    2571 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195319    2571 flags.go:64] FLAG: --help="false"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195323    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195326    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195330    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195333    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195336    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195340    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195343    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195347    2571 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195355    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 23:18:20.201725 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195358    2571 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195361    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195388    2571 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195392    2571 flags.go:64] FLAG: --kube-reserved=""
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195395    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195399    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195402    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195405    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195408    2571 flags.go:64] FLAG: --lock-file=""
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195411    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195414    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195418    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195424    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195427    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195430    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195433    2571 flags.go:64] FLAG: --logging-format="text"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195437    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195440    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195443    2571 flags.go:64] FLAG: --manifest-url=""
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195446    2571 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195455    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195458    2571 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195463    2571 flags.go:64] FLAG: --max-pods="110"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195466    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195470    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 23:18:20.202269 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195473    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195477    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195480    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195483    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195487    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195497    2571 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195502    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195505    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195509    2571 flags.go:64] FLAG: --pod-cidr=""
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195512    2571 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195515    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195518    2571 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195521    2571 flags.go:64] FLAG: --port="10250"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195524    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195527    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-064b3e15d49d28a10"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195531    2571 flags.go:64] FLAG: --qos-reserved=""
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195534    2571 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195537    2571 flags.go:64] FLAG: --register-node="true"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195541    2571 flags.go:64] FLAG: --register-with-taints=""
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195544    2571 flags.go:64] FLAG: --registry-burst="10"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195547    2571 flags.go:64] FLAG: --registry-qps="5"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195550    2571 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195553    2571 flags.go:64] FLAG: --reserved-memory=""
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195558    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195561    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195564    2571 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 23:18:20.202880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195567    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195570    2571 flags.go:64] FLAG: --runonce="false"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195573    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195576    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195580    2571 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195583    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195586    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195590    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195593    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195597    2571 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195600    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195604    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195608    2571 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195611    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195615    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195618    2571 flags.go:64] FLAG: --system-cgroups=""
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195621    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195626    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195629    2571 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195632    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195637    2571 flags.go:64] FLAG: --tls-min-version=""
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195640    2571 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195643    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195646    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195649    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 23:18:20.203499 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195652    2571 flags.go:64] FLAG: --v="2"
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195657    2571 flags.go:64] FLAG: --version="false"
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195661    2571 flags.go:64] FLAG: --vmodule=""
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195666    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.195669    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195835    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195839    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195842    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195845    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195847    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195850    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195853    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195856    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195858    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195862    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195864    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195867    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195870    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195875    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195880    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:18:20.204095 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195884    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195887    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195890    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195893    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195896    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195898    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195901    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195904    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195906    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195909    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195912    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195914    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195917    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195919    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195922    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195924    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195927    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195930    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195932    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195935    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:18:20.204603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195938    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195940    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195943    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195946    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195949    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195952    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195954    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195959    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195962    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195965    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195968    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195971    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195974    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195976    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195979    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195981    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195984    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195987    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195991    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:18:20.205067 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195995    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.195998    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196001    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196003    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196006    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196008    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196011    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196014    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196016    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196019    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196022    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196024    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196027    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196029    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196032    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196035    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196037    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196040    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196042    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:18:20.205559 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196045    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196047    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196051    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196054    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196057    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196060    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196062    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196065    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196067    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196070    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196073    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196075    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196078    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196081    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196083    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196086    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196089    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196091    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196094    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196096    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:18:20.206035 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196099    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196101    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196104    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196106    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196109    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196111    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196114    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196117    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196119    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196122    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196125    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196128    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196130    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.196156    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196210    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196215    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:18:20.206523 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196221    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196224    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196227    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196229    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196232    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196235    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196237    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196240    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196243    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196245    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196248    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196251    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196254    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196257    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196260    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196262    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196265    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196267    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196270    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196273    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:18:20.206901 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196275    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196279    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196284    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196288    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196290    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196293    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196296    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196299    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196301    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196304    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196307    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196309    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196314    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196316    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196319    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196322    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196324    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196327    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196330    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:18:20.207377 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196332    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196335    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196337    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196340    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196343    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196345    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196348    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196350    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196353    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196356    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196359    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196362    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196364    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196385    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196389    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196391    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196394    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196397    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196401    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196403    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:18:20.207842 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196406    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196409    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196411    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196414    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196417    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196420    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196424    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196427    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196430    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196433    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196435    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196438    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196441    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196443    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196446    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196449    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196451    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196454    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196457    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196460    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:18:20.208324 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196462    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196465    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196468    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196470    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196473    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196475    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196478    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196480    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196484    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196488    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196493    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196496    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196499    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196502    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196505    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196508    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196510    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196514    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196517    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196520    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:18:20.208817 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196523    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196526    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196528    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196531    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.196533    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.203937    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.203959    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.205033    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.205045    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205113    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205119    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205123    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205126    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205129    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205132    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205135    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205138    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205140    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205143    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:18:20.209295 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205146    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205149    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205151    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205154    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205157    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205160    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205162    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205165    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205168    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205171    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205173    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205176    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205178    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205181    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205183    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205186    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205188    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205191    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205193    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:18:20.209923 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205202    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205206    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205210    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205215    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205219    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205221    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205224    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205227    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205229    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205232    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205234    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205237    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205240    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205243    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205245    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205248    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205251    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205255    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205258    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:18:20.210603 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205261    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205264    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205266    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205269    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205271    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205274    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205277    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205280    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205283    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205285    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205292    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205294    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205297    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205300    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205308    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205311    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205313    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205316    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205318    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205321    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:18:20.211084 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205324    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205326    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205329    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205331    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205334    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205337    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205339    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205342    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205350    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205353    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205356    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205358    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205361    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205363    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205379    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205382    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205385    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205387    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205390    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205393    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:18:20.211808 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205396    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205398    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205401    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205403    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205406    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205409    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205412    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205420    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205423    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205426    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205428    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205431    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205433    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205436    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205438    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205441    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205443    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:18:20.212307 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205446    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.205479    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205677    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205683    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205686    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205690    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205693    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205695    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205698    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205700    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205703    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205705    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205708    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205711    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205713    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205716    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:18:20.212740 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205718    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205721    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205723    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205726    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205729    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205732    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205734    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205743    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205746    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205748    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205751    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205753    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205756    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205759    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205761    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205764    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205768    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205772    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205775    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:18:20.213110 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205778    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205781    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205785    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205787    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205790    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205793    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205795    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205798    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205800    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205803    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205805    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205808    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205811    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205813    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205815    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205818    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205821    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205824    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205826    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205829    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:18:20.213577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205831    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205839    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205842    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205844    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205847    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205849    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205852    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205854    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205857    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205860    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205862    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205865    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205867    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205870    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205872    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205875    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205878    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205880    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205883    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205885    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:18:20.214042 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205888    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205890    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205893    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205895    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205898    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205900    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205903    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205905    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205908    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205910    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205913    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205915    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205918    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205920    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205929    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205932    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205934    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205937    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205939    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205942    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:18:20.214681 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205944    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205947    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205950    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205952    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205955    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205959    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205962    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205966    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205969    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205971    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205974    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205976    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.205979    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.206003    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206039    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206044    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:18:20.215186 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206047    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206050    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206053    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206055    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206058    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206061    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206063    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206066    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206068    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206071    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206073    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206081    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206084    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206087    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206090    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206092    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206095    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206097    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206100    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206102    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:18:20.215592 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206104    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206107    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206110    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206113    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206115    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206118    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206120    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206123    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206125    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206128    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206130    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206132    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206135    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206138    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206140    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206142    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206145    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206148    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206150    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206153    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:18:20.216150 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206155    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206158    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206160    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206163    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206171    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206174    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206176    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206179    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206182    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206184    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206188    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206192    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206196    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206198    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206201    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206203    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206206    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206209    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206212    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206214    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:18:20.216723 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206217    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206220    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206222    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206225    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206227    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206230    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206232    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206235    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206237    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206240    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206243    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206245    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206248    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206251    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206253    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206255    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206258    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206266    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206269    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:18:20.217193 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206271    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206274    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206276    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206279    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206282    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206290    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206293    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206295    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206298    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206302    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206305    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206308    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206311    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206315    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206340    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206344    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206347    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206350    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206353    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:18:20.217656 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206355    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206358    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206360    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206363    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206379    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:20.206382    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.206439    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.208490    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.211009    2571 server.go:1014] "Starting client certificate rotation"
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.211519    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.211563    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:18:20.218305 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.215548    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:18:20.218642 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.218398    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:18:20.229001 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.228980    2571 log.go:25] "Validated CRI v1 runtime API"
Oct 30 23:18:20.232713 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.232697    2571 log.go:25] "Validated CRI v1 image API"
Oct 30 23:18:20.234972 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.234936    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 23:18:20.242274 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.242249    2571 fs.go:135] Filesystem UUIDs: map[7059bbec-926b-43d2-9f92-19a42ac5ce25:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cad86b51-2eac-488c-adc0-7c712f6bb863:/dev/nvme0n1p4]
Oct 30 23:18:20.242384 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.242273    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 23:18:20.247894 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.247763    2571 manager.go:217] Machine: {Timestamp:2026-10-30 23:18:20.246328855 +0000 UTC m=+0.345351093 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098653 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20f21b94f6479a0827079abdb090e8 SystemUUID:ec20f21b-94f6-479a-0827-079abdb090e8 BootID:ee1ef1d6-0d50-4082-a5fc-cdfa7085208d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:86:86:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:86:86:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:d1:ec:d5:c1:ce Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 23:18:20.247894 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.247890    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 23:18:20.248055 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.248042    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 23:18:20.250423 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.250393    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 23:18:20.250558 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.250423    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-83.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 23:18:20.250602 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.250561    2571 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 23:18:20.250602 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.250569    2571 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 23:18:20.250602 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.250600    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:18:20.250689 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.250613    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:18:20.251651 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.251640    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 23:18:20.255472 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.255454    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 23:18:20.255822 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.255810    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 23:18:20.255942 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.255933    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 23:18:20.255981 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.255971    2571 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 23:18:20.256008 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.255985    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 23:18:20.256008 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.255996    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 23:18:20.256008 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.256005    2571 kubelet.go:405] "Adding apiserver pod source"
Oct 30 23:18:20.256110 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.256013    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 23:18:20.256430 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.256411    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:18:20.259305 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.259287    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 23:18:20.265248 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.265219    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:18:20.266073 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266060    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 23:18:20.266115 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266091    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 23:18:20.266259 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266249    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 23:18:20.266294 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266264    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 23:18:20.266294 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266270    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 23:18:20.266294 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266275    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 23:18:20.266294 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266281    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 23:18:20.266294 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266286    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 23:18:20.266935 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266926    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 23:18:20.266967 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266936    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 23:18:20.266967 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266945    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 23:18:20.266967 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266950    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 23:18:20.266967 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266960    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 23:18:20.266967 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.266968    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 23:18:20.267646 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.267637    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 23:18:20.267646 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.267646    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 23:18:20.271636 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.271617    2571 server.go:1263] "Started kubelet"
Oct 30 23:18:20.271734 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.271688    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 23:18:20.272106 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.272057    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 23:18:20.272206 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.272124    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 23:18:20.272342 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.272307    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-83.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 23:18:20.272446 ip-10-0-134-83 systemd[1]: Started Kubernetes Kubelet.
Oct 30 23:18:20.273655 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.273606    2571 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 23:18:20.273655 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.273651    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 23:18:20.277555 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.277534    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-83.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 23:18:20.277555 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.277540    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 23:18:20.278108 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.278091    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 23:18:20.278195 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.277026    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-83.us-east-2.compute.internal.18e371e3e70e916b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-83.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:18:20.271595883 +0000 UTC m=+0.370618121,LastTimestamp:2026-10-30 23:18:20.271595883 +0000 UTC m=+0.370618121,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-83.us-east-2.compute.internal,}"
Oct 30 23:18:20.279456 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.279440    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tjd9j"
Oct 30 23:18:20.280613 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.280595    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 23:18:20.281210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.281196    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 23:18:20.281300 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.281283    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:20.281980 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.281966    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 23:18:20.282073 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.281968    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 23:18:20.282073 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.282000    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 23:18:20.282147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.282094    2571 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 23:18:20.282147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.282102    2571 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 23:18:20.282147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.282121    2571 factory.go:153] Registering CRI-O factory
Oct 30 23:18:20.282147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.282132    2571 factory.go:223] Registration of the crio container factory successfully
Oct 30 23:18:20.282267 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.282173    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 23:18:20.282267 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.282180    2571 factory.go:55] Registering systemd factory
Oct 30 23:18:20.282267 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.282185    2571 factory.go:223] Registration of the systemd container factory successfully
Oct 30 23:18:20.282267 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.282195    2571 factory.go:103] Registering Raw factory
Oct 30 23:18:20.282267 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.282204    2571 manager.go:1199] Started watching for new ooms in manager
Oct 30 23:18:20.283450 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.283434    2571 manager.go:319] Starting recovery of all containers
Oct 30 23:18:20.283818 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.283710    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-83.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 23:18:20.288596 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.288567    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 23:18:20.290603 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.290578    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 23:18:20.297065 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.297041    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tjd9j"
Oct 30 23:18:20.297428 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.297412    2571 manager.go:324] Recovery completed
Oct 30 23:18:20.301880 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.301867    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:18:20.303817 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.303801    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:18:20.303896 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.303829    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:18:20.303896 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.303839    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:18:20.304406 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.304395    2571 cpu_manager.go:226] "Starting" policy="none"
Oct 30 23:18:20.304406 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.304404    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 23:18:20.304506 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.304407    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:18:20.304506 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.304421    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 23:18:20.305105 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.305088    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:18:20.305145 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.305126    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:18:20.305145 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.305143    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:18:20.306242 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.306169    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-83.us-east-2.compute.internal.18e371e3e8fa3319  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-83.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-83.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:18:20.303815449 +0000 UTC m=+0.402837687,LastTimestamp:2026-10-30 23:18:20.303815449 +0000 UTC m=+0.402837687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-83.us-east-2.compute.internal,}"
Oct 30 23:18:20.306337 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.306309    2571 policy_none.go:50] "Start"
Oct 30 23:18:20.306337 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.306322    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 23:18:20.306337 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.306333    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 23:18:20.308292 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.308281    2571 policy_none.go:44] "Start"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.351101    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.355554    2571 manager.go:341] "Starting Device Plugin manager"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.355582    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.355592    2571 server.go:87] "Starting device plugin registration server"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.355902    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.355916    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.356010    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.356158    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.356165    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.356606    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.356653    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.358501    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.358522    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.358539    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.358569    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 23:18:20.365196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.362313    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:18:20.456272 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.456226    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:18:20.457241 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.457219    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:18:20.457362 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.457260    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:18:20.457362 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.457274    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:18:20.457362 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.457300    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.459491 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.459473    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"]
Oct 30 23:18:20.459539 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.459527    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:18:20.460274 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.460259    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:18:20.460360 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.460288    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:18:20.460360 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.460300    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:18:20.462518 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.462506    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:18:20.462669 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.462656    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.462708 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.462683    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:18:20.463254 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.463235    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:18:20.463316 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.463250    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:18:20.463316 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.463264    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:18:20.463316 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.463273    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:18:20.463316 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.463277    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:18:20.463316 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.463283    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:18:20.464231 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.464218    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.464269 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.464238    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-83.us-east-2.compute.internal\": node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:20.465342 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.465327    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.465402 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.465359    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:18:20.466154 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.466135    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:18:20.466246 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.466173    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:18:20.466246 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.466188    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:18:20.478510 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.478493    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:20.483321 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.483301    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.483429 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.483334    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.483429 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.483361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17ddb1b82b212964d7163812b102551c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"17ddb1b82b212964d7163812b102551c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.485642 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.485626    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found" node="ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.488440 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.488424    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found" node="ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.579233 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.579205    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:20.583498 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.583475    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.583590 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.583507    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.583590 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.583525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17ddb1b82b212964d7163812b102551c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"17ddb1b82b212964d7163812b102551c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.583590 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.583576    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.583590 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.583581    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.583721 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.583622    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17ddb1b82b212964d7163812b102551c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"17ddb1b82b212964d7163812b102551c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.679922 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.679857    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:20.780581 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.780541    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:20.791018 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.790994    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.793163 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:20.793144    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:20.881256 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.881218    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:20.981729 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:20.981640    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:21.082094 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:21.082056    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:21.182580 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:21.182540    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:21.210854 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.210823    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 23:18:21.211293 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.211009    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 23:18:21.282761 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:21.282670    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:21.303273 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.303236    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:13:20 +0000 UTC" deadline="2028-06-19 18:39:13.973582851 +0000 UTC"
Oct 30 23:18:21.303273 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.303265    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14347h20m52.670320561s"
Oct 30 23:18:21.332077 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.332048    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:18:21.383090 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:21.383054    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:21.483554 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:21.483511    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:18:21.522901 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.522873    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:18:21.526315 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:21.526289    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ddb1b82b212964d7163812b102551c.slice/crio-c872e40ef133a0dec402980ced6bc6ab41f1aa50063cf9fb7d567c68b53b1693 WatchSource:0}: Error finding container c872e40ef133a0dec402980ced6bc6ab41f1aa50063cf9fb7d567c68b53b1693: Status 404 returned error can't find the container with id c872e40ef133a0dec402980ced6bc6ab41f1aa50063cf9fb7d567c68b53b1693
Oct 30 23:18:21.526621 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:21.526603    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a12773649a9a35ac251a27d3fd9a87d.slice/crio-3773ebbbf252b2ecc7853e5584eccf71a5caabac272710021c931fcc1dc77ee7 WatchSource:0}: Error finding container 3773ebbbf252b2ecc7853e5584eccf71a5caabac272710021c931fcc1dc77ee7: Status 404 returned error can't find the container with id 3773ebbbf252b2ecc7853e5584eccf71a5caabac272710021c931fcc1dc77ee7
Oct 30 23:18:21.533513 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.533498    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:18:21.543727 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.543711    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:18:21.581483 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.581455    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:21.599386 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.599344    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:18:21.600443 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.600430    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:18:21.608971 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:21.608954    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:18:22.073458 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.073418    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:18:22.257991 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.257954    2571 apiserver.go:52] "Watching apiserver"
Oct 30 23:18:22.264807 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.264749    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 23:18:22.265152 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.265129    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-7g6g7","openshift-dns/node-resolver-czzsf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal","openshift-multus/multus-9x8lb","openshift-network-diagnostics/network-check-target-vzvtj","kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal","openshift-image-registry/node-ca-prlzd","openshift-multus/multus-additional-cni-plugins-mlh9x","openshift-network-operator/iptables-alerter-c7np6","openshift-ovn-kubernetes/ovnkube-node-kptck","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc","openshift-multus/network-metrics-daemon-vb7vz","kube-system/konnectivity-agent-l94rx","openshift-cluster-node-tuning-operator/tuned-ckpdq"]
Oct 30 23:18:22.267894 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.267866    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.271012 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.270399    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.274785 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.272740    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 23:18:22.274785 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.272800    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 23:18:22.274785 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.272885    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 23:18:22.274785 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.272921    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4bbvk\""
Oct 30 23:18:22.274785 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.273507    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.274785 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.273817    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 23:18:22.274785 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.274059    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 23:18:22.274785 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.274129    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kc52k\""
Oct 30 23:18:22.274785 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.274186    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 23:18:22.278459 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.275994    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.278459 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.277039    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 23:18:22.278459 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.277103    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:18:22.278459 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.277236    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n9xrj\""
Oct 30 23:18:22.278716 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.278558    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.279486 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.279458    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-82xnp\""
Oct 30 23:18:22.279597 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.279579    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 23:18:22.279665 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.279617    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 23:18:22.279804 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.279777    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 23:18:22.279885 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.279879    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 23:18:22.280833 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.280812    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:22.280942 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.280898    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:22.282111 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.281698    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 23:18:22.283762 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.283729    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 23:18:22.283922 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.283906    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.286668 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.286642    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:22.288868 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.288849    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.291662 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.291640    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.294223 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.294205    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:22.294337 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.294283    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:22.296970 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.296944    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 23:18:22.297086 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.297067    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:22.297171 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.297141    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:22.302144 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.302120    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-socket-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.302249 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.302159    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-registration-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.304493 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.302321    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8rhz\" (UniqueName: \"kubernetes.io/projected/fb61c865-8cb6-46d3-a802-82c8abfb691d-kube-api-access-r8rhz\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.304595 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.304539    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-tuned\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.304649 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.304585    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfq2\" (UniqueName: \"kubernetes.io/projected/54da46ee-38c8-441e-a219-552cd01c7340-kube-api-access-dlfq2\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.304838 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.304663    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.304916 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.304866    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-cnibin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.304975 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.304927    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-sys-fs\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.304975 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.304962    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-cnibin\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.305139 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.305122    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-netns\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.305264 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.305216    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-hostroot\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.305560 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.305528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-host\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.305731 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.305638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-serviceca\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.305811 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.305760    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.305876 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.305836    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-var-lib-kubelet\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.305876 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.305871    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-tmp\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.305987 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.305905    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-system-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.305987 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.305942    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-kubelet\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.306107 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.305987    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-etc-kubernetes\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.306107 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.306107 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306051    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-device-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.306107 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306086    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-kubernetes\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.306322 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306125    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-lib-modules\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.306322 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306162    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9fpl\" (UniqueName: \"kubernetes.io/projected/d012abb7-7079-484a-8f4c-46b9c8bed1a4-kube-api-access-q9fpl\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.306322 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.306322 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306240    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/093e503b-974c-4918-9a88-0de16b2aa06d-iptables-alerter-script\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.306322 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306291    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-multus-daemon-config\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.306627 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306331    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-modprobe-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.306627 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306368    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-sys\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.306627 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-system-cni-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.306627 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.306627 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306506    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc2ft\" (UniqueName: \"kubernetes.io/projected/99889cd4-347f-46b6-938f-0886271bb04d-kube-api-access-nc2ft\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:22.306627 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306538    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-socket-dir-parent\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.306627 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-k8s-cni-cncf-io\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.306627 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306598    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-conf-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.306627 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306627    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306651    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-conf\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306681    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-cni-binary-copy\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306710    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zh6\" (UniqueName: \"kubernetes.io/projected/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-kube-api-access-k8zh6\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306741    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-run\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306791    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-host\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306820    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-bin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306850    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-multus\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306884    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-multus-certs\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306910    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86llh\" (UniqueName: \"kubernetes.io/projected/83690100-e937-4dba-80e5-b5af4b7e507d-kube-api-access-86llh\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.306973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.307015    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysconfig\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.307053    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-systemd\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.307104 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.307084    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:22.308969 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.307119    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/093e503b-974c-4918-9a88-0de16b2aa06d-host-slash\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.308969 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.307150    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr8cj\" (UniqueName: \"kubernetes.io/projected/093e503b-974c-4918-9a88-0de16b2aa06d-kube-api-access-wr8cj\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.323991 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.323939    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2r5pj"
Oct 30 23:18:22.335482 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.335420    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2r5pj"
Oct 30 23:18:22.363337 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.363274    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal" event={"ID":"8a12773649a9a35ac251a27d3fd9a87d","Type":"ContainerStarted","Data":"3773ebbbf252b2ecc7853e5584eccf71a5caabac272710021c931fcc1dc77ee7"}
Oct 30 23:18:22.364670 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.364635    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal" event={"ID":"17ddb1b82b212964d7163812b102551c","Type":"ContainerStarted","Data":"c872e40ef133a0dec402980ced6bc6ab41f1aa50063cf9fb7d567c68b53b1693"}
Oct 30 23:18:22.383633 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.383594    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 23:18:22.384575 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384550    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 23:18:22.384729 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384557    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 23:18:22.384729 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384692    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 23:18:22.384729 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384717    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qmkhv\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384743    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384759    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384777    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384818    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384819    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384834    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384862    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384889    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384892    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jt8kr\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384827    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k8dwv\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384895    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 23:18:22.384926 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384902    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:18:22.385528 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384947    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 23:18:22.385528 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384983    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lwvlw\""
Oct 30 23:18:22.385528 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.384997    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qwtvf\""
Oct 30 23:18:22.407798 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.407758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-bin\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.407798 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.407798    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-config\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.408030 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.407826    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-kubelet-config\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:22.408030 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.407849    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/093e503b-974c-4918-9a88-0de16b2aa06d-iptables-alerter-script\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.408030 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.407870    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-system-cni-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.408030 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.407889    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2ft\" (UniqueName: \"kubernetes.io/projected/99889cd4-347f-46b6-938f-0886271bb04d-kube-api-access-nc2ft\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:22.408030 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.407916    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-socket-dir-parent\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.408030 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.407942    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-conf-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.408030 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.407968    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.408030 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.407997    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:22.408030 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/321dcdee-cb90-4de4-8c18-39120efda8e4-konnectivity-ca\") pod \"konnectivity-agent-l94rx\" (UID: \"321dcdee-cb90-4de4-8c18-39120efda8e4\") " pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408051    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-systemd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408117    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-run\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-bin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408169    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86llh\" (UniqueName: \"kubernetes.io/projected/83690100-e937-4dba-80e5-b5af4b7e507d-kube-api-access-86llh\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408200    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fea866f2-91be-418e-8a5f-dc609e36cfd5-hosts-file\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408225    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-kubelet\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408241    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-env-overrides\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/093e503b-974c-4918-9a88-0de16b2aa06d-host-slash\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408282    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408297    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea866f2-91be-418e-8a5f-dc609e36cfd5-tmp-dir\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408313    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f805291-3eab-499d-aed8-577d64cf5c8c-ovn-node-metrics-cert\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408329    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-registration-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-tuned\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408379    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfq2\" (UniqueName: \"kubernetes.io/projected/54da46ee-38c8-441e-a219-552cd01c7340-kube-api-access-dlfq2\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408399    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-netns\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.408475 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408423    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-host\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408475    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408504    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-sys-fs\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408522    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-hostroot\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408548    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-var-lib-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408575    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-system-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408610    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/321dcdee-cb90-4de4-8c18-39120efda8e4-agent-certs\") pod \"konnectivity-agent-l94rx\" (UID: \"321dcdee-cb90-4de4-8c18-39120efda8e4\") " pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408626    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-log-socket\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408665    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-dbus\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408687    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408722    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-device-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408749    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-kubernetes\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408776    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-lib-modules\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fpl\" (UniqueName: \"kubernetes.io/projected/d012abb7-7079-484a-8f4c-46b9c8bed1a4-kube-api-access-q9fpl\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408810    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.409210 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-multus-daemon-config\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-multus-certs\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408886    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-modprobe-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408902    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-sys\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408962    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-k8s-cni-cncf-io\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.408987    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zh6\" (UniqueName: \"kubernetes.io/projected/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-kube-api-access-k8zh6\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409006    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-slash\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409030    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-etc-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409056    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-node-log\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409081    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-conf\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409104    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-cni-binary-copy\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-host\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409174    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-multus\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409201    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scsx6\" (UniqueName: \"kubernetes.io/projected/fea866f2-91be-418e-8a5f-dc609e36cfd5-kube-api-access-scsx6\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-ovn\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409253    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.410064 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-netd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409309    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-script-lib\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm86f\" (UniqueName: \"kubernetes.io/projected/6f805291-3eab-499d-aed8-577d64cf5c8c-kube-api-access-wm86f\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409368    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-etc-kubernetes\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409422    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysconfig\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-systemd\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409527    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8cj\" (UniqueName: \"kubernetes.io/projected/093e503b-974c-4918-9a88-0de16b2aa06d-kube-api-access-wr8cj\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-socket-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409578    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8rhz\" (UniqueName: \"kubernetes.io/projected/fb61c865-8cb6-46d3-a802-82c8abfb691d-kube-api-access-r8rhz\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409605    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-cnibin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409659    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-cnibin\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409685    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-serviceca\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409703    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-systemd-units\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409726    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-netns\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.410912 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409754    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-var-lib-kubelet\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409779    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-tmp\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409807    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-kubelet\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.409910    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-kubelet\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.410583    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/093e503b-974c-4918-9a88-0de16b2aa06d-iptables-alerter-script\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.410620    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-lib-modules\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.410692    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-system-cni-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.411003    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-socket-dir-parent\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.411060    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-conf-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.411115    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.411271    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.411321    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-run\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.411364    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-bin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.411543    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.411662    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-sys\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.411675    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/093e503b-974c-4918-9a88-0de16b2aa06d-host-slash\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.411747 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.411756    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-registration-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.412555 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.412012    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 23:18:22.412555 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.412194    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-multus-daemon-config\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.412555 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.412339    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.412555 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.412354    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-etc-kubernetes\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.412555 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.412416    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysconfig\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.412555 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.412540    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-system-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.412855 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.412584    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.412855 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.412646    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-systemd\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.412855 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.412689    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-device-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.412855 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.412687    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-host\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.412855 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.412745    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:22.412855 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.412811    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs podName:99889cd4-347f-46b6-938f-0886271bb04d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:22.912790762 +0000 UTC m=+3.011813005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs") pod "network-metrics-daemon-vb7vz" (UID: "99889cd4-347f-46b6-938f-0886271bb04d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:22.413197 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413174    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-socket-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.413354 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413328    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-cnibin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.413406 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413374    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-netns\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.413484 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413408    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-k8s-cni-cncf-io\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.413484 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413427    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-host\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.413594 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413540    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-multus\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.413651 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413633    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-kubernetes\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.414550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413778    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-sys-fs\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.414550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413871    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-serviceca\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.414550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413916    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-modprobe-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.414550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413921    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-multus-certs\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.414550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413930    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-conf\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.414550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413973    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-hostroot\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.414550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.413989    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.414550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.414025    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-cnibin\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.414550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.414063    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-var-lib-kubelet\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.414550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.414506    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-cni-binary-copy\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.415078 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.414599    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.415078 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.414596    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.424162 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.416513    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-tuned\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.424162 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.417292    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-tmp\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.425891 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.425857    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fpl\" (UniqueName: \"kubernetes.io/projected/d012abb7-7079-484a-8f4c-46b9c8bed1a4-kube-api-access-q9fpl\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.427403 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.427373    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfq2\" (UniqueName: \"kubernetes.io/projected/54da46ee-38c8-441e-a219-552cd01c7340-kube-api-access-dlfq2\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.427664 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.427614    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8cj\" (UniqueName: \"kubernetes.io/projected/093e503b-974c-4918-9a88-0de16b2aa06d-kube-api-access-wr8cj\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.428597 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.428568    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zh6\" (UniqueName: \"kubernetes.io/projected/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-kube-api-access-k8zh6\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.428704 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.428581    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8rhz\" (UniqueName: \"kubernetes.io/projected/fb61c865-8cb6-46d3-a802-82c8abfb691d-kube-api-access-r8rhz\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.428781 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.428759    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86llh\" (UniqueName: \"kubernetes.io/projected/83690100-e937-4dba-80e5-b5af4b7e507d-kube-api-access-86llh\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.429241 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.429220    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2ft\" (UniqueName: \"kubernetes.io/projected/99889cd4-347f-46b6-938f-0886271bb04d-kube-api-access-nc2ft\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:22.510885 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.510846    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:22.510885 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.510895    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-systemd-units\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.510921    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-netns\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.510945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-bin\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.510964    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-config\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.510985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-kubelet-config\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:22.511147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511012    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:22.511147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511035    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/321dcdee-cb90-4de4-8c18-39120efda8e4-konnectivity-ca\") pod \"konnectivity-agent-l94rx\" (UID: \"321dcdee-cb90-4de4-8c18-39120efda8e4\") " pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:22.511147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511055    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-systemd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511080    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fea866f2-91be-418e-8a5f-dc609e36cfd5-hosts-file\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.511147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511105    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-kubelet\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511147 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-env-overrides\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511157    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea866f2-91be-418e-8a5f-dc609e36cfd5-tmp-dir\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511180    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f805291-3eab-499d-aed8-577d64cf5c8c-ovn-node-metrics-cert\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511213    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511243    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-var-lib-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511269    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/321dcdee-cb90-4de4-8c18-39120efda8e4-agent-certs\") pod \"konnectivity-agent-l94rx\" (UID: \"321dcdee-cb90-4de4-8c18-39120efda8e4\") " pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511325    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-log-socket\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-dbus\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511389    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-slash\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-etc-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-node-log\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511498    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scsx6\" (UniqueName: \"kubernetes.io/projected/fea866f2-91be-418e-8a5f-dc609e36cfd5-kube-api-access-scsx6\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511531    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-ovn\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511583    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-netd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-script-lib\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.511692 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.511660    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm86f\" (UniqueName: \"kubernetes.io/projected/6f805291-3eab-499d-aed8-577d64cf5c8c-kube-api-access-wm86f\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.512546 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.512257    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:22.512546 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.512374    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret podName:a6a30dfd-0560-4e49-9890-07cb7f8c8bcd nodeName:}" failed. No retries permitted until 2026-10-30 23:18:23.012353169 +0000 UTC m=+3.111375414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret") pod "global-pull-secret-syncer-7g6g7" (UID: "a6a30dfd-0560-4e49-9890-07cb7f8c8bcd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:22.512546 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.512455    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-systemd-units\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.512705 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.512543    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-var-lib-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.512705 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.512577    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-node-log\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.512705 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.512593    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-netns\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.512705 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.512633    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-bin\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.512705 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.512645    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.512705 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.512664    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-netd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.512997 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.512709    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-kubelet-config\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:22.512997 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.512766    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-ovn\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.513141 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.513118    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-slash\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.513214 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.513180    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-log-socket\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.513333 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.513318    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-dbus\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:22.513429 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.513369    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.513586 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.513471    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-etc-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.513586 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.513534    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fea866f2-91be-418e-8a5f-dc609e36cfd5-hosts-file\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.513703 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.513591    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-systemd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.514068 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.513865    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.514068 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.513926    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-kubelet\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.514217 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.514094    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-config\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.514217 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.514143    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/321dcdee-cb90-4de4-8c18-39120efda8e4-konnectivity-ca\") pod \"konnectivity-agent-l94rx\" (UID: \"321dcdee-cb90-4de4-8c18-39120efda8e4\") " pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:22.514372 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.514344    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-env-overrides\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.514579 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.514496    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-script-lib\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.514999 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.514710    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea866f2-91be-418e-8a5f-dc609e36cfd5-tmp-dir\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.516970 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.516946    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f805291-3eab-499d-aed8-577d64cf5c8c-ovn-node-metrics-cert\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.520293 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.520234    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/321dcdee-cb90-4de4-8c18-39120efda8e4-agent-certs\") pod \"konnectivity-agent-l94rx\" (UID: \"321dcdee-cb90-4de4-8c18-39120efda8e4\") " pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:22.523583 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.523560    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:18:22.523683 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.523586    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:18:22.523683 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.523601    2571 projected.go:196] Error preparing data for projected volume kube-api-access-pzwqv for pod openshift-network-diagnostics/network-check-target-vzvtj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:22.523683 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.523670    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv podName:f223a1d7-7a59-45a5-970f-deecf721ff88 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:23.023651827 +0000 UTC m=+3.122674075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pzwqv" (UniqueName: "kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv") pod "network-check-target-vzvtj" (UID: "f223a1d7-7a59-45a5-970f-deecf721ff88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:22.526288 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.526261    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scsx6\" (UniqueName: \"kubernetes.io/projected/fea866f2-91be-418e-8a5f-dc609e36cfd5-kube-api-access-scsx6\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.526464 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.526416    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm86f\" (UniqueName: \"kubernetes.io/projected/6f805291-3eab-499d-aed8-577d64cf5c8c-kube-api-access-wm86f\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.588593 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.588506    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:18:22.599472 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:22.599419    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6748f5ad_4980_4e4b_b448_bdc7f50e9e57.slice/crio-2536bb12e29523bf461b1335af480a645e70794c776fdc06e8c40670ddba4810 WatchSource:0}: Error finding container 2536bb12e29523bf461b1335af480a645e70794c776fdc06e8c40670ddba4810: Status 404 returned error can't find the container with id 2536bb12e29523bf461b1335af480a645e70794c776fdc06e8c40670ddba4810
Oct 30 23:18:22.605196 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.605160    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:18:22.617022 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.616949    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:18:22.617288 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:22.617253    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb61c865_8cb6_46d3_a802_82c8abfb691d.slice/crio-e4fdacd55354bb13b061bd099d0c6db4fd948027c7d7371f4445e3a428d2fd73 WatchSource:0}: Error finding container e4fdacd55354bb13b061bd099d0c6db4fd948027c7d7371f4445e3a428d2fd73: Status 404 returned error can't find the container with id e4fdacd55354bb13b061bd099d0c6db4fd948027c7d7371f4445e3a428d2fd73
Oct 30 23:18:22.622566 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.622538    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9x8lb"
Oct 30 23:18:22.626898 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:22.626566    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd012abb7_7079_484a_8f4c_46b9c8bed1a4.slice/crio-d3071e334b3811a28ea02846090842f5145965ff6508e2b29f73ba41616f01f2 WatchSource:0}: Error finding container d3071e334b3811a28ea02846090842f5145965ff6508e2b29f73ba41616f01f2: Status 404 returned error can't find the container with id d3071e334b3811a28ea02846090842f5145965ff6508e2b29f73ba41616f01f2
Oct 30 23:18:22.634069 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:22.634040    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83690100_e937_4dba_80e5_b5af4b7e507d.slice/crio-3e21826aebd0c6f3feb0242e77c2a4ec37013076e09349310115e507556f3cd6 WatchSource:0}: Error finding container 3e21826aebd0c6f3feb0242e77c2a4ec37013076e09349310115e507556f3cd6: Status 404 returned error can't find the container with id 3e21826aebd0c6f3feb0242e77c2a4ec37013076e09349310115e507556f3cd6
Oct 30 23:18:22.641742 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.641711    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:18:22.644921 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.644888    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:18:22.647163 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.647140    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:22.656628 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.656603    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:18:22.656738 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:22.656706    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093e503b_974c_4918_9a88_0de16b2aa06d.slice/crio-a25a3119963b4699abcb98d9a01469e04c4d900e809719ece8011ceec2ccdf93 WatchSource:0}: Error finding container a25a3119963b4699abcb98d9a01469e04c4d900e809719ece8011ceec2ccdf93: Status 404 returned error can't find the container with id a25a3119963b4699abcb98d9a01469e04c4d900e809719ece8011ceec2ccdf93
Oct 30 23:18:22.659856 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:22.659659    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54da46ee_38c8_441e_a219_552cd01c7340.slice/crio-ea556f7c8aa60bbf9382abb1db8cb12713f4b57e482ce96d73e41efbe8004481 WatchSource:0}: Error finding container ea556f7c8aa60bbf9382abb1db8cb12713f4b57e482ce96d73e41efbe8004481: Status 404 returned error can't find the container with id ea556f7c8aa60bbf9382abb1db8cb12713f4b57e482ce96d73e41efbe8004481
Oct 30 23:18:22.668577 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:22.668547    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea866f2_91be_418e_8a5f_dc609e36cfd5.slice/crio-3b706a7844f9e86078423190b2efcfadff1c2c63edf30fec9c714f17be65ac13 WatchSource:0}: Error finding container 3b706a7844f9e86078423190b2efcfadff1c2c63edf30fec9c714f17be65ac13: Status 404 returned error can't find the container with id 3b706a7844f9e86078423190b2efcfadff1c2c63edf30fec9c714f17be65ac13
Oct 30 23:18:22.684610 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.684582    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:22.699118 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:22.699072    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f805291_3eab_499d_aed8_577d64cf5c8c.slice/crio-9a7428cb5ca52fe976082c38a46ce1d0362db76ac71c075eeeadd9c0b4bd7916 WatchSource:0}: Error finding container 9a7428cb5ca52fe976082c38a46ce1d0362db76ac71c075eeeadd9c0b4bd7916: Status 404 returned error can't find the container with id 9a7428cb5ca52fe976082c38a46ce1d0362db76ac71c075eeeadd9c0b4bd7916
Oct 30 23:18:22.915778 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:22.915691    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:22.915940 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.915856    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:22.915940 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:22.915931    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs podName:99889cd4-347f-46b6-938f-0886271bb04d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:23.915911238 +0000 UTC m=+4.014933467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs") pod "network-metrics-daemon-vb7vz" (UID: "99889cd4-347f-46b6-938f-0886271bb04d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:23.016509 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.016475    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:23.016751 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:23.016591    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:23.016751 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:23.016653    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret podName:a6a30dfd-0560-4e49-9890-07cb7f8c8bcd nodeName:}" failed. No retries permitted until 2026-10-30 23:18:24.016636615 +0000 UTC m=+4.115658842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret") pod "global-pull-secret-syncer-7g6g7" (UID: "a6a30dfd-0560-4e49-9890-07cb7f8c8bcd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:23.118034 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.117882    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:23.118210 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:23.118057    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:18:23.118210 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:23.118081    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:18:23.118210 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:23.118092    2571 projected.go:196] Error preparing data for projected volume kube-api-access-pzwqv for pod openshift-network-diagnostics/network-check-target-vzvtj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:23.118210 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:23.118158    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv podName:f223a1d7-7a59-45a5-970f-deecf721ff88 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:24.11812924 +0000 UTC m=+4.217151471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzwqv" (UniqueName: "kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv") pod "network-check-target-vzvtj" (UID: "f223a1d7-7a59-45a5-970f-deecf721ff88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:23.339023 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.338852    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:13:22 +0000 UTC" deadline="2028-04-15 05:05:26.976763615 +0000 UTC"
Oct 30 23:18:23.339023 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.338906    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12773h47m3.637861639s"
Oct 30 23:18:23.396413 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.396029    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-czzsf" event={"ID":"fea866f2-91be-418e-8a5f-dc609e36cfd5","Type":"ContainerStarted","Data":"3b706a7844f9e86078423190b2efcfadff1c2c63edf30fec9c714f17be65ac13"}
Oct 30 23:18:23.401693 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.401658    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c7np6" event={"ID":"093e503b-974c-4918-9a88-0de16b2aa06d","Type":"ContainerStarted","Data":"a25a3119963b4699abcb98d9a01469e04c4d900e809719ece8011ceec2ccdf93"}
Oct 30 23:18:23.413704 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.413660    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ckpdq" event={"ID":"d012abb7-7079-484a-8f4c-46b9c8bed1a4","Type":"ContainerStarted","Data":"d3071e334b3811a28ea02846090842f5145965ff6508e2b29f73ba41616f01f2"}
Oct 30 23:18:23.420162 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.420128    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kptck" event={"ID":"6f805291-3eab-499d-aed8-577d64cf5c8c","Type":"ContainerStarted","Data":"9a7428cb5ca52fe976082c38a46ce1d0362db76ac71c075eeeadd9c0b4bd7916"}
Oct 30 23:18:23.424870 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.424815    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l94rx" event={"ID":"321dcdee-cb90-4de4-8c18-39120efda8e4","Type":"ContainerStarted","Data":"0b2edeb6a09fc60695979195b8fdc000a2525c795ff20fd8342e7421e5d9568d"}
Oct 30 23:18:23.426693 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.426630    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" event={"ID":"54da46ee-38c8-441e-a219-552cd01c7340","Type":"ContainerStarted","Data":"ea556f7c8aa60bbf9382abb1db8cb12713f4b57e482ce96d73e41efbe8004481"}
Oct 30 23:18:23.432467 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.431833    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9x8lb" event={"ID":"83690100-e937-4dba-80e5-b5af4b7e507d","Type":"ContainerStarted","Data":"3e21826aebd0c6f3feb0242e77c2a4ec37013076e09349310115e507556f3cd6"}
Oct 30 23:18:23.443498 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.443330    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc" event={"ID":"fb61c865-8cb6-46d3-a802-82c8abfb691d","Type":"ContainerStarted","Data":"e4fdacd55354bb13b061bd099d0c6db4fd948027c7d7371f4445e3a428d2fd73"}
Oct 30 23:18:23.446729 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.446614    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-prlzd" event={"ID":"6748f5ad-4980-4e4b-b448-bdc7f50e9e57","Type":"ContainerStarted","Data":"2536bb12e29523bf461b1335af480a645e70794c776fdc06e8c40670ddba4810"}
Oct 30 23:18:23.926469 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:23.926022    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:23.926469 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:23.926340    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:23.926697 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:23.926552    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs podName:99889cd4-347f-46b6-938f-0886271bb04d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:25.926526723 +0000 UTC m=+6.025548955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs") pod "network-metrics-daemon-vb7vz" (UID: "99889cd4-347f-46b6-938f-0886271bb04d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:24.027560 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.026912    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:24.027560 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:24.027083    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:24.027560 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:24.027151    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret podName:a6a30dfd-0560-4e49-9890-07cb7f8c8bcd nodeName:}" failed. No retries permitted until 2026-10-30 23:18:26.027131117 +0000 UTC m=+6.126153364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret") pod "global-pull-secret-syncer-7g6g7" (UID: "a6a30dfd-0560-4e49-9890-07cb7f8c8bcd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:24.127508 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.127460    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:24.127736 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:24.127718    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:18:24.127806 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:24.127744    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:18:24.127806 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:24.127758    2571 projected.go:196] Error preparing data for projected volume kube-api-access-pzwqv for pod openshift-network-diagnostics/network-check-target-vzvtj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:24.127915 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:24.127821    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv podName:f223a1d7-7a59-45a5-970f-deecf721ff88 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:26.127800808 +0000 UTC m=+6.226823052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzwqv" (UniqueName: "kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv") pod "network-check-target-vzvtj" (UID: "f223a1d7-7a59-45a5-970f-deecf721ff88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:24.339959 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.339882    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:13:22 +0000 UTC" deadline="2028-07-13 02:28:58.39789514 +0000 UTC"
Oct 30 23:18:24.339959 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.339950    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14907h10m34.057950639s"
Oct 30 23:18:24.368526 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.365828    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:24.368526 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:24.365965    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:24.368526 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.366401    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:24.368526 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:24.366535    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:24.368526 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.366611    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:24.368526 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:24.366688    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:24.468248 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.468016    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal" event={"ID":"17ddb1b82b212964d7163812b102551c","Type":"ContainerStarted","Data":"3e9614c52dac7e4219c8782a5b26c2ec2b1f9a9393d8566219388b2b36f72509"}
Oct 30 23:18:24.478990 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.478778    2571 generic.go:356] "Generic (PLEG): container finished" podID="8a12773649a9a35ac251a27d3fd9a87d" containerID="6b61e5375beb0951b541e9a7492aa816f83e596d283e080f387833221aa92244" exitCode=0
Oct 30 23:18:24.478990 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.478837    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal" event={"ID":"8a12773649a9a35ac251a27d3fd9a87d","Type":"ContainerDied","Data":"6b61e5375beb0951b541e9a7492aa816f83e596d283e080f387833221aa92244"}
Oct 30 23:18:24.486237 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:24.486170    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal" podStartSLOduration=2.486153382 podStartE2EDuration="2.486153382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:24.485759091 +0000 UTC m=+4.584781342" watchObservedRunningTime="2026-10-30 23:18:24.486153382 +0000 UTC m=+4.585175634"
Oct 30 23:18:25.486611 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:25.486525    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal" event={"ID":"8a12773649a9a35ac251a27d3fd9a87d","Type":"ContainerStarted","Data":"d4a48773e7bf24e594cecfece2cd3bd7383baee652b4184cbf968ac181b1b832"}
Oct 30 23:18:25.510488 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:25.510417    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal" podStartSLOduration=3.510398693 podStartE2EDuration="3.510398693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:25.510195317 +0000 UTC m=+5.609217570" watchObservedRunningTime="2026-10-30 23:18:25.510398693 +0000 UTC m=+5.609420945"
Oct 30 23:18:25.946022 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:25.945978    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:25.946318 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:25.946249    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:25.946390 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:25.946372    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs podName:99889cd4-347f-46b6-938f-0886271bb04d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:29.946352213 +0000 UTC m=+10.045374454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs") pod "network-metrics-daemon-vb7vz" (UID: "99889cd4-347f-46b6-938f-0886271bb04d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:26.047854 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:26.047488    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:26.047854 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:26.047676    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:26.047854 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:26.047744    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret podName:a6a30dfd-0560-4e49-9890-07cb7f8c8bcd nodeName:}" failed. No retries permitted until 2026-10-30 23:18:30.047724649 +0000 UTC m=+10.146746888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret") pod "global-pull-secret-syncer-7g6g7" (UID: "a6a30dfd-0560-4e49-9890-07cb7f8c8bcd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:26.148948 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:26.148144    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:26.148948 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:26.148316    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:18:26.148948 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:26.148341    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:18:26.148948 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:26.148355    2571 projected.go:196] Error preparing data for projected volume kube-api-access-pzwqv for pod openshift-network-diagnostics/network-check-target-vzvtj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:26.148948 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:26.148438    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv podName:f223a1d7-7a59-45a5-970f-deecf721ff88 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:30.14839749 +0000 UTC m=+10.247419742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzwqv" (UniqueName: "kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv") pod "network-check-target-vzvtj" (UID: "f223a1d7-7a59-45a5-970f-deecf721ff88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:26.359714 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:26.359638    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:26.359843 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:26.359784    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:26.360363 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:26.360245    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:26.360491 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:26.360364    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:26.360491 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:26.360416    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:26.360607 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:26.360518    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:26.493455 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:26.493357    2571 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="68f2eef015e849ec06c4c1f1ce36fdb66b27c56768f02967d54cd527566fce94" exitCode=0
Oct 30 23:18:26.494512 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:26.494483    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" event={"ID":"54da46ee-38c8-441e-a219-552cd01c7340","Type":"ContainerDied","Data":"68f2eef015e849ec06c4c1f1ce36fdb66b27c56768f02967d54cd527566fce94"}
Oct 30 23:18:28.360769 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:28.359043    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:28.360769 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:28.359176    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:28.360769 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:28.359250    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:28.360769 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:28.359321    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:28.360769 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:28.359397    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:28.360769 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:28.359467    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:28.506938 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:28.506857    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l94rx" event={"ID":"321dcdee-cb90-4de4-8c18-39120efda8e4","Type":"ContainerStarted","Data":"389c0bf15a6251d48d04c7f991d2ceea1f00c5bdb791cd24fffab579b26429b6"}
Oct 30 23:18:28.511631 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:28.511603    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-prlzd" event={"ID":"6748f5ad-4980-4e4b-b448-bdc7f50e9e57","Type":"ContainerStarted","Data":"4e5ce87576225128fc95f8cda2997e80a30bc75030982e7944ed19e277d5af9e"}
Oct 30 23:18:28.546262 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:28.546203    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l94rx" podStartSLOduration=3.654641159 podStartE2EDuration="8.546184034s" totalImagesPullingTime="4.891542875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:28.525394776 +0000 UTC m=+8.624417024" watchObservedRunningTime="2026-10-30 23:18:28.546184034 +0000 UTC m=+8.645206282"
Oct 30 23:18:28.546784 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:28.546740    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-prlzd" podStartSLOduration=2.901396035 podStartE2EDuration="8.546724266s" totalImagesPullingTime="5.645328231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:28.545722033 +0000 UTC m=+8.644744281" watchObservedRunningTime="2026-10-30 23:18:28.546724266 +0000 UTC m=+8.645746505"
Oct 30 23:18:29.518879 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:29.518842    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc" event={"ID":"fb61c865-8cb6-46d3-a802-82c8abfb691d","Type":"ContainerStarted","Data":"365455ad7606a0f56bcae743189b93986d61e6d184ed7ae9124304738c72c103"}
Oct 30 23:18:29.984884 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:29.984801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:29.985056 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:29.984994    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:29.985123 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:29.985059    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs podName:99889cd4-347f-46b6-938f-0886271bb04d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:37.985040528 +0000 UTC m=+18.084062767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs") pod "network-metrics-daemon-vb7vz" (UID: "99889cd4-347f-46b6-938f-0886271bb04d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:30.085289 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:30.085238    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:30.085464 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:30.085444    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:30.085541 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:30.085507    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret podName:a6a30dfd-0560-4e49-9890-07cb7f8c8bcd nodeName:}" failed. No retries permitted until 2026-10-30 23:18:38.085486798 +0000 UTC m=+18.184509043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret") pod "global-pull-secret-syncer-7g6g7" (UID: "a6a30dfd-0560-4e49-9890-07cb7f8c8bcd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:30.186103 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:30.186061    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:30.186263 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:30.186238    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:18:30.186263 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:30.186252    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:18:30.186263 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:30.186262    2571 projected.go:196] Error preparing data for projected volume kube-api-access-pzwqv for pod openshift-network-diagnostics/network-check-target-vzvtj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:30.186448 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:30.186304    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv podName:f223a1d7-7a59-45a5-970f-deecf721ff88 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:38.186292707 +0000 UTC m=+18.285314945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzwqv" (UniqueName: "kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv") pod "network-check-target-vzvtj" (UID: "f223a1d7-7a59-45a5-970f-deecf721ff88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:30.381362 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:30.380521    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:30.381362 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:30.380646    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:30.381362 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:30.381069    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:30.381362 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:30.381190    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:30.381362 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:30.381245    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:30.381362 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:30.381324    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:30.461189 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:30.461153    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 23:18:30.526896 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:30.526863    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc" event={"ID":"fb61c865-8cb6-46d3-a802-82c8abfb691d","Type":"ContainerStarted","Data":"f214c9e70045297cf233070a461364a743c4657eb947a0a31cf675464b20eeef"}
Oct 30 23:18:31.372239 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:31.372118    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:18:30.461180012Z","UUID":"47ef6aaf-29c7-4f6a-be4d-0a2474308402","Handler":null,"Name":"","Endpoint":""}
Oct 30 23:18:31.374024 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:31.373995    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 23:18:31.374273 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:31.374035    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 23:18:31.531353 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:31.531316    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" event={"ID":"54da46ee-38c8-441e-a219-552cd01c7340","Type":"ContainerStarted","Data":"8ffa6672b6e7c2da51eaac2f0588ec4328342b3dc27560eea6a22f4ce507e4dd"}
Oct 30 23:18:31.534088 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:31.534062    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9x8lb" event={"ID":"83690100-e937-4dba-80e5-b5af4b7e507d","Type":"ContainerStarted","Data":"7f1c9b089c75626c7343efb4c15ccfb072f4974099bc7b209857dbd92d58b3c1"}
Oct 30 23:18:31.535724 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:31.535678    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-czzsf" event={"ID":"fea866f2-91be-418e-8a5f-dc609e36cfd5","Type":"ContainerStarted","Data":"61198289ba7e9b33b811eef85986638800918c6293067faa6fbe17cf53864818"}
Oct 30 23:18:31.574518 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:31.574458    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9x8lb" podStartSLOduration=3.606150367 podStartE2EDuration="11.574440392s" totalImagesPullingTime="7.968290025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:31.574110203 +0000 UTC m=+11.673132452" watchObservedRunningTime="2026-10-30 23:18:31.574440392 +0000 UTC m=+11.673462640"
Oct 30 23:18:32.360286 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.359466    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:32.360286 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:32.359809    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:32.360286 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.360171    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:32.360286 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:32.360240    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:32.360569 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.360363    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:32.360569 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:32.360513    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:32.540354 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.540234    2571 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="8ffa6672b6e7c2da51eaac2f0588ec4328342b3dc27560eea6a22f4ce507e4dd" exitCode=0
Oct 30 23:18:32.540354 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.540269    2571 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="175e5c97682cf317cc5f1451e875e955097dbb3ff888b352d63bc692478686d2" exitCode=0
Oct 30 23:18:32.540856 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.540352    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" event={"ID":"54da46ee-38c8-441e-a219-552cd01c7340","Type":"ContainerDied","Data":"8ffa6672b6e7c2da51eaac2f0588ec4328342b3dc27560eea6a22f4ce507e4dd"}
Oct 30 23:18:32.540856 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.540436    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" event={"ID":"54da46ee-38c8-441e-a219-552cd01c7340","Type":"ContainerDied","Data":"175e5c97682cf317cc5f1451e875e955097dbb3ff888b352d63bc692478686d2"}
Oct 30 23:18:32.543541 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.543509    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc" event={"ID":"fb61c865-8cb6-46d3-a802-82c8abfb691d","Type":"ContainerStarted","Data":"ff085285914e2f425d9fd543130cb77a9b092936e35e124863d2b5efbd00dcb7"}
Oct 30 23:18:32.565291 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.565196    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-czzsf" podStartSLOduration=4.24836883 podStartE2EDuration="12.565183507s" totalImagesPullingTime="8.316814677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:31.590604412 +0000 UTC m=+11.689626659" watchObservedRunningTime="2026-10-30 23:18:32.565183507 +0000 UTC m=+12.664205754"
Oct 30 23:18:32.587678 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.587575    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc" podStartSLOduration=4.030865929 podStartE2EDuration="12.587559834s" totalImagesPullingTime="8.556693905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:32.587118412 +0000 UTC m=+12.686140660" watchObservedRunningTime="2026-10-30 23:18:32.587559834 +0000 UTC m=+12.686582082"
Oct 30 23:18:32.959224 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.959180    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:32.959943 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:32.959926    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:33.547352 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:33.547268    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c7np6" event={"ID":"093e503b-974c-4918-9a88-0de16b2aa06d","Type":"ContainerStarted","Data":"55fb262f868f8a71201176a00ff0c8c2864825ba8878ab89f1d131e21e9a9c84"}
Oct 30 23:18:33.548754 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:33.548719    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ckpdq" event={"ID":"d012abb7-7079-484a-8f4c-46b9c8bed1a4","Type":"ContainerStarted","Data":"a44327eebb4da11cd6b13893b42bb17626a3679f8417ae4180356b352f1ab2e1"}
Oct 30 23:18:33.564121 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:33.564072    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c7np6" podStartSLOduration=5.084000667 podStartE2EDuration="13.564059135s" totalImagesPullingTime="8.480058468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:33.563615379 +0000 UTC m=+13.662637627" watchObservedRunningTime="2026-10-30 23:18:33.564059135 +0000 UTC m=+13.663081382"
Oct 30 23:18:33.583795 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:33.583732    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ckpdq" podStartSLOduration=2.944134776 podStartE2EDuration="13.583716782s" totalImagesPullingTime="10.639582006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:33.58340529 +0000 UTC m=+13.682427539" watchObservedRunningTime="2026-10-30 23:18:33.583716782 +0000 UTC m=+13.682739029"
Oct 30 23:18:34.359881 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:34.359845    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:34.360046 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:34.359885    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:34.360103 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:34.360039    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:34.360103 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:34.360073    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:34.360221 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:34.360188    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:34.360298 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:34.360274    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:34.553948 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:34.553910    2571 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="2dbf9ad29af42554fdb8623420b9516127f9ffd601b5b3251eab47e7d6cba325" exitCode=0
Oct 30 23:18:34.554564 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:34.554003    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" event={"ID":"54da46ee-38c8-441e-a219-552cd01c7340","Type":"ContainerDied","Data":"2dbf9ad29af42554fdb8623420b9516127f9ffd601b5b3251eab47e7d6cba325"}
Oct 30 23:18:36.359735 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:36.359701    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:36.359735 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:36.359734    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:36.360459 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:36.359701    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:36.360459 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:36.359816    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:36.360459 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:36.359865    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:36.360459 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:36.359944    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:36.561456 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:36.561186    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" event={"ID":"54da46ee-38c8-441e-a219-552cd01c7340","Type":"ContainerStarted","Data":"0c063b6c4e119cbd063700a9906fc1ff78b5846e5c43aaae165f35a3cf04a549"}
Oct 30 23:18:37.566194 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:37.566159    2571 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="0c063b6c4e119cbd063700a9906fc1ff78b5846e5c43aaae165f35a3cf04a549" exitCode=0
Oct 30 23:18:37.566194 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:37.566199    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" event={"ID":"54da46ee-38c8-441e-a219-552cd01c7340","Type":"ContainerDied","Data":"0c063b6c4e119cbd063700a9906fc1ff78b5846e5c43aaae165f35a3cf04a549"}
Oct 30 23:18:38.046476 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.046450    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:38.046605 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.046586    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:38.046654 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.046645    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs podName:99889cd4-347f-46b6-938f-0886271bb04d nodeName:}" failed. No retries permitted until 2026-10-30 23:18:54.046630936 +0000 UTC m=+34.145653164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs") pod "network-metrics-daemon-vb7vz" (UID: "99889cd4-347f-46b6-938f-0886271bb04d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:18:38.147500 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.147464    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:38.147637 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.147602    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:38.147695 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.147663    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret podName:a6a30dfd-0560-4e49-9890-07cb7f8c8bcd nodeName:}" failed. No retries permitted until 2026-10-30 23:18:54.147645902 +0000 UTC m=+34.246668127 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret") pod "global-pull-secret-syncer-7g6g7" (UID: "a6a30dfd-0560-4e49-9890-07cb7f8c8bcd") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:18:38.248801 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.248751    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:38.248988 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.248888    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:18:38.248988 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.248911    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:18:38.248988 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.248920    2571 projected.go:196] Error preparing data for projected volume kube-api-access-pzwqv for pod openshift-network-diagnostics/network-check-target-vzvtj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:38.248988 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.248967    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv podName:f223a1d7-7a59-45a5-970f-deecf721ff88 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:54.248954832 +0000 UTC m=+34.347977058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzwqv" (UniqueName: "kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv") pod "network-check-target-vzvtj" (UID: "f223a1d7-7a59-45a5-970f-deecf721ff88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:18:38.359446 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.359343    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:38.359612 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.359354    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:38.359612 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.359492    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:38.359612 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.359354    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:38.359612 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.359541    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:38.359612 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:38.359601    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:38.570869 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.570677    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kptck" event={"ID":"6f805291-3eab-499d-aed8-577d64cf5c8c","Type":"ContainerStarted","Data":"fd87505d53b651c00db90880d3b930671d959553b620ca018b6a614217ca2dca"}
Oct 30 23:18:38.571312 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.570912    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kptck" event={"ID":"6f805291-3eab-499d-aed8-577d64cf5c8c","Type":"ContainerStarted","Data":"d905fb02368c2c93611c6262852ed0f9179dd8b46381b6da1e1c1b7eed822ce4"}
Oct 30 23:18:38.571312 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.570948    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kptck" event={"ID":"6f805291-3eab-499d-aed8-577d64cf5c8c","Type":"ContainerStarted","Data":"f13b5d69bcabc7b6c4caa5e20e0f1c5434f2b9e598ed9318a780e904ef2403f9"}
Oct 30 23:18:38.571312 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.570978    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kptck" event={"ID":"6f805291-3eab-499d-aed8-577d64cf5c8c","Type":"ContainerStarted","Data":"925d8cae05c743e9dc2ea1c0cec6f2d4febe1d299299b747b203f5a7aa77e771"}
Oct 30 23:18:38.571312 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.571009    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kptck" event={"ID":"6f805291-3eab-499d-aed8-577d64cf5c8c","Type":"ContainerStarted","Data":"6d1ee88dddd14ca01e22d77921a9c2ec1a8bf24b3f7945bd1c8a99095a1017e1"}
Oct 30 23:18:38.571312 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.571052    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kptck" event={"ID":"6f805291-3eab-499d-aed8-577d64cf5c8c","Type":"ContainerStarted","Data":"0834af7a233255282110e7cfee724edefb5780e929b6b8fb1380ad9bd6584e5b"}
Oct 30 23:18:38.573695 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.573674    2571 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="614a8c7530fec0a54ec384cda510569cab2f97d5b9d25b9db7a76cde2633d46a" exitCode=0
Oct 30 23:18:38.573772 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:38.573712    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" event={"ID":"54da46ee-38c8-441e-a219-552cd01c7340","Type":"ContainerDied","Data":"614a8c7530fec0a54ec384cda510569cab2f97d5b9d25b9db7a76cde2633d46a"}
Oct 30 23:18:39.578492 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:39.578445    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" event={"ID":"54da46ee-38c8-441e-a219-552cd01c7340","Type":"ContainerStarted","Data":"7490cb55cd6b711fe879eb5b62228815c55cd5c22af2ec4a434f29efcb44e3bf"}
Oct 30 23:18:39.604841 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:39.604794    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mlh9x" podStartSLOduration=8.26984763 podStartE2EDuration="19.60478066s" totalImagesPullingTime="11.33493303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:39.603346204 +0000 UTC m=+19.702368451" watchObservedRunningTime="2026-10-30 23:18:39.60478066 +0000 UTC m=+19.703802906"
Oct 30 23:18:40.360102 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:40.360069    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:40.360285 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:40.360204    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:40.360285 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:40.360229    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:40.360285 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:40.360263    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:40.360485 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:40.360311    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:40.360485 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:40.360407    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:40.583356 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:40.583269    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kptck" event={"ID":"6f805291-3eab-499d-aed8-577d64cf5c8c","Type":"ContainerStarted","Data":"9ab77accb22bafddf522ea63912641c4fbb29514427efcf7a2d6100d82fa3560"}
Oct 30 23:18:42.360659 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:42.359788    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:42.360659 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:42.359937    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:42.360659 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:42.360336    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:42.360659 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:42.360460    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:42.360659 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:42.360520    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:42.360659 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:42.360623    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:42.594187 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:42.594105    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kptck" event={"ID":"6f805291-3eab-499d-aed8-577d64cf5c8c","Type":"ContainerStarted","Data":"97d57e7f906336b8541f9cf93050ca0e2e10ca1588c85fd1c3a610cbb2306f57"}
Oct 30 23:18:42.594425 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:42.594337    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:42.610945 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:42.610921    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:42.665741 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:42.665686    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kptck" podStartSLOduration=7.62127631 podStartE2EDuration="22.665674807s" totalImagesPullingTime="15.044398497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:42.664291595 +0000 UTC m=+22.763313842" watchObservedRunningTime="2026-10-30 23:18:42.665674807 +0000 UTC m=+22.764697054"
Oct 30 23:18:43.596308 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.596283    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:18:43.596742 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.596721    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:43.614597 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.614573    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:43.624063 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.624034    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:43.624234 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.624219    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:18:43.624644 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.624620    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:18:43.991558 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.991308    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7g6g7"]
Oct 30 23:18:43.991705 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.991603    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:43.991705 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:43.991690    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:43.994307 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.994280    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzvtj"]
Oct 30 23:18:43.994518 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.994414    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:43.994585 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:43.994515    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:43.994800 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.994783    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vb7vz"]
Oct 30 23:18:43.994906 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:43.994890    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:43.995001 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:43.994984    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:44.601846 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:44.601748    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:18:45.359811 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:45.359774    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:45.359811 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:45.359806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:45.360036 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:45.359782    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:45.360036 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:45.359920    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:45.360140 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:45.360043    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:45.360221 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:45.360191    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:45.604359 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:45.604331    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:18:47.359739 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:47.359700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:47.360226 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:47.359700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:47.360226 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:47.359831    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:47.360226 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:47.359700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:47.360226 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:47.359891    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:47.360226 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:47.359963    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:49.362015 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:49.359245    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:49.362015 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:49.359435    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:49.362015 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:49.359461    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzvtj" podUID="f223a1d7-7a59-45a5-970f-deecf721ff88"
Oct 30 23:18:49.362015 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:49.359478    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:49.362015 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:49.360684    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb7vz" podUID="99889cd4-347f-46b6-938f-0886271bb04d"
Oct 30 23:18:49.362015 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:49.360829    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7g6g7" podUID="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd"
Oct 30 23:18:49.846069 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:49.846040    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:49.846290 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:49.846275    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:18:49.858683 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:49.858657    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:18:50.131536 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.131462    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeReady"
Oct 30 23:18:50.131687 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.131587    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 23:18:50.181716 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.181679    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b4l22"]
Oct 30 23:18:50.184681 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.184666    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.188417 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.188392    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lgv9k"]
Oct 30 23:18:50.191189 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.191169    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:18:50.194303 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.194277    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 23:18:50.195310 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.195287    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b4l22"]
Oct 30 23:18:50.201033 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.200971    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lgv9k"]
Oct 30 23:18:50.292084 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.292050    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 23:18:50.292254 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.292138    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gskv9\""
Oct 30 23:18:50.292254 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.292167    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zcrv2\""
Oct 30 23:18:50.292254 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.292179    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 23:18:50.295420 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.295402    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 23:18:50.295470 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.295431    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 23:18:50.329441 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.329400    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.329649 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.329481    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ca7641-8711-4a48-86e0-a59f3ca8e498-tmp-dir\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.329649 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.329534    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4rz\" (UniqueName: \"kubernetes.io/projected/82ca7641-8711-4a48-86e0-a59f3ca8e498-kube-api-access-jf4rz\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.329649 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.329603    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82ca7641-8711-4a48-86e0-a59f3ca8e498-config-volume\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.329649 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.329624    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-cert\") pod \"ingress-canary-lgv9k\" (UID: \"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d\") " pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:18:50.329821 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.329652    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x55vs\" (UniqueName: \"kubernetes.io/projected/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-kube-api-access-x55vs\") pod \"ingress-canary-lgv9k\" (UID: \"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d\") " pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:18:50.430894 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.430811    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.430894 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.430862    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ca7641-8711-4a48-86e0-a59f3ca8e498-tmp-dir\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.430894 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.430879    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4rz\" (UniqueName: \"kubernetes.io/projected/82ca7641-8711-4a48-86e0-a59f3ca8e498-kube-api-access-jf4rz\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.430894 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.430896    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82ca7641-8711-4a48-86e0-a59f3ca8e498-config-volume\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.431485 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.430913    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-cert\") pod \"ingress-canary-lgv9k\" (UID: \"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d\") " pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:18:50.431485 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.430931    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x55vs\" (UniqueName: \"kubernetes.io/projected/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-kube-api-access-x55vs\") pod \"ingress-canary-lgv9k\" (UID: \"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d\") " pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:18:50.431485 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:50.430977    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:18:50.431485 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:50.431046    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls podName:82ca7641-8711-4a48-86e0-a59f3ca8e498 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:50.931025589 +0000 UTC m=+31.030047822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls") pod "dns-default-b4l22" (UID: "82ca7641-8711-4a48-86e0-a59f3ca8e498") : secret "dns-default-metrics-tls" not found
Oct 30 23:18:50.431485 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.431352    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ca7641-8711-4a48-86e0-a59f3ca8e498-tmp-dir\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.431634 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.431536    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82ca7641-8711-4a48-86e0-a59f3ca8e498-config-volume\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.435245 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.435133    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-cert\") pod \"ingress-canary-lgv9k\" (UID: \"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d\") " pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:18:50.444146 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.444126    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4rz\" (UniqueName: \"kubernetes.io/projected/82ca7641-8711-4a48-86e0-a59f3ca8e498-kube-api-access-jf4rz\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.444463 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.444443    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x55vs\" (UniqueName: \"kubernetes.io/projected/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-kube-api-access-x55vs\") pod \"ingress-canary-lgv9k\" (UID: \"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d\") " pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:18:50.503809 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.503770    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:18:50.797713 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.797682    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lgv9k"]
Oct 30 23:18:50.801315 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:50.801292    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fa85d1_6827_43dd_a947_71d6ebc1ce6d.slice/crio-a0122039b3f65e344e980f1d72bcee94afa767fe506e91be5d820c40daf6e64d WatchSource:0}: Error finding container a0122039b3f65e344e980f1d72bcee94afa767fe506e91be5d820c40daf6e64d: Status 404 returned error can't find the container with id a0122039b3f65e344e980f1d72bcee94afa767fe506e91be5d820c40daf6e64d
Oct 30 23:18:50.935556 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:50.935516    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:50.935728 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:50.935668    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:18:50.935774 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:50.935732    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls podName:82ca7641-8711-4a48-86e0-a59f3ca8e498 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:51.935717201 +0000 UTC m=+32.034739426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls") pod "dns-default-b4l22" (UID: "82ca7641-8711-4a48-86e0-a59f3ca8e498") : secret "dns-default-metrics-tls" not found
Oct 30 23:18:51.359120 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.359088    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:51.359120 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.359128    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:51.359120 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.359127    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:51.363262 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.363209    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:18:51.363399 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.363259    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:18:51.363473 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.363413    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:18:51.363512 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.363474    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9tmfn\""
Oct 30 23:18:51.363512 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.363504    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djdr2\""
Oct 30 23:18:51.363582 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.363504    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:18:51.363582 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.363540    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2nz9z\""
Oct 30 23:18:51.618528 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.618447    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lgv9k" event={"ID":"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d","Type":"ContainerStarted","Data":"a0122039b3f65e344e980f1d72bcee94afa767fe506e91be5d820c40daf6e64d"}
Oct 30 23:18:51.942011 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:51.941862    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:51.942175 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:51.942054    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:18:51.942175 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:51.942146    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls podName:82ca7641-8711-4a48-86e0-a59f3ca8e498 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:53.942125586 +0000 UTC m=+34.041147815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls") pod "dns-default-b4l22" (UID: "82ca7641-8711-4a48-86e0-a59f3ca8e498") : secret "dns-default-metrics-tls" not found
Oct 30 23:18:53.955387 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:53.955292    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:53.955829 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:53.955487    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:18:53.955829 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:53.955570    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls podName:82ca7641-8711-4a48-86e0-a59f3ca8e498 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:57.955550311 +0000 UTC m=+38.054572557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls") pod "dns-default-b4l22" (UID: "82ca7641-8711-4a48-86e0-a59f3ca8e498") : secret "dns-default-metrics-tls" not found
Oct 30 23:18:54.056520 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.056485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:18:54.056671 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:54.056602    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 23:18:54.056671 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:54.056653    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs podName:99889cd4-347f-46b6-938f-0886271bb04d nodeName:}" failed. No retries permitted until 2026-10-30 23:19:26.056640989 +0000 UTC m=+66.155663214 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs") pod "network-metrics-daemon-vb7vz" (UID: "99889cd4-347f-46b6-938f-0886271bb04d") : secret "metrics-daemon-secret" not found
Oct 30 23:18:54.157341 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.157316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:54.159769 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.159747    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:54.258388 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.258344    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:54.261063 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.261040    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:54.384658 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.384624    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:54.426794 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.426771    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:18:54.627479 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.627444    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lgv9k" event={"ID":"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d","Type":"ContainerStarted","Data":"c31dae6d308796cb56fd44062602dcbf202208ffe929007e9691e336158b129c"}
Oct 30 23:18:54.644996 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.644879    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lgv9k" podStartSLOduration=1.306397807 podStartE2EDuration="4.644861873s" totalImagesPullingTime="3.338464066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:54.644359847 +0000 UTC m=+34.743382093" watchObservedRunningTime="2026-10-30 23:18:54.644861873 +0000 UTC m=+34.743884120"
Oct 30 23:18:54.684988 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.684834    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzvtj"]
Oct 30 23:18:54.689130 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:54.689106    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf223a1d7_7a59_45a5_970f_deecf721ff88.slice/crio-d2a4554309cb996246b363eb61c06dd68655660435c9d7a2efd7c7c351226321 WatchSource:0}: Error finding container d2a4554309cb996246b363eb61c06dd68655660435c9d7a2efd7c7c351226321: Status 404 returned error can't find the container with id d2a4554309cb996246b363eb61c06dd68655660435c9d7a2efd7c7c351226321
Oct 30 23:18:54.719518 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:54.719493    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7g6g7"]
Oct 30 23:18:54.722230 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:54.722204    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a30dfd_0560_4e49_9890_07cb7f8c8bcd.slice/crio-f98d49b9315b6f83e51dd5fff61e432771d5ec4221c2318a8d136a4ccc3f90a5 WatchSource:0}: Error finding container f98d49b9315b6f83e51dd5fff61e432771d5ec4221c2318a8d136a4ccc3f90a5: Status 404 returned error can't find the container with id f98d49b9315b6f83e51dd5fff61e432771d5ec4221c2318a8d136a4ccc3f90a5
Oct 30 23:18:55.633646 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.633600    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7g6g7" event={"ID":"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd","Type":"ContainerStarted","Data":"f98d49b9315b6f83e51dd5fff61e432771d5ec4221c2318a8d136a4ccc3f90a5"}
Oct 30 23:18:55.634955 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.634928    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzvtj" event={"ID":"f223a1d7-7a59-45a5-970f-deecf721ff88","Type":"ContainerStarted","Data":"d2a4554309cb996246b363eb61c06dd68655660435c9d7a2efd7c7c351226321"}
Oct 30 23:18:55.733523 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.733481    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wk72x"]
Oct 30 23:18:55.755997 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.755961    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vlbq7-jsbgj"]
Oct 30 23:18:55.756164 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.756057    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.769859 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.769832    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wk72x"]
Oct 30 23:18:55.770019 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.769887    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vlbq7-jsbgj"]
Oct 30 23:18:55.770084 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.770021    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.856735 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.856698    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 23:18:55.856735 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.856733    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 23:18:55.857041 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.856711    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 23:18:55.857041 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.856843    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 23:18:55.857041 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.856854    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 23:18:55.857041 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.857024    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z9rpn\""
Oct 30 23:18:55.857284 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.857161    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mx9rq\""
Oct 30 23:18:55.859685 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.859657    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 23:18:55.868758 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.868729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-archives-path\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.868924 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.868768    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mffv\" (UniqueName: \"kubernetes.io/projected/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-kube-api-access-5mffv\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.868924 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.868799    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-service-ca-bundle\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.869033 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.868974    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.869084 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.869057    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-crio-socket\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.869140 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.869117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-data-volume\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.869192 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.869138    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkvp\" (UniqueName: \"kubernetes.io/projected/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-api-access-jpkvp\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.869192 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.869156    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-trusted-ca-bundle\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.869339 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.869325    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.970428 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.970332    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-crio-socket\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.970428 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.970424    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-data-volume\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.970643 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.970452    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkvp\" (UniqueName: \"kubernetes.io/projected/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-api-access-jpkvp\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.970643 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.970478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-trusted-ca-bundle\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.970643 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.970521    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.970643 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.970548    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-archives-path\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.970643 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.970566    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mffv\" (UniqueName: \"kubernetes.io/projected/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-kube-api-access-5mffv\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.970643 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.970583    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-service-ca-bundle\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.970643 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.970601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.970643 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.970605    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-crio-socket\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.970940 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:55.970708    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:18:55.970940 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:55.970759    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:56.470744169 +0000 UTC m=+36.569766430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:18:55.971223 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.971076    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-data-volume\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:55.971223 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:55.971118    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:18:55.971223 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:55.971184    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:56.471166365 +0000 UTC m=+36.570188607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : configmap "kube-rbac-proxy" not found
Oct 30 23:18:55.971454 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.971435    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-archives-path\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.971550 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.971525    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-service-ca-bundle\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.971874 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.971850    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-trusted-ca-bundle\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.983952 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.983924    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mffv\" (UniqueName: \"kubernetes.io/projected/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-kube-api-access-5mffv\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:55.984120 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:55.984052    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkvp\" (UniqueName: \"kubernetes.io/projected/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-api-access-jpkvp\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:56.125816 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.125776    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:18:56.434058 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.434018    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vlbq7-jsbgj"]
Oct 30 23:18:56.439624 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:56.439584    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481ae6f7_3ce8_4964_a9e2_60d26b2c6b06.slice/crio-8eca9237ef2e72f8839e1d94ae7f7bed67f833431a8b64cc60fb4ce53d8acfe0 WatchSource:0}: Error finding container 8eca9237ef2e72f8839e1d94ae7f7bed67f833431a8b64cc60fb4ce53d8acfe0: Status 404 returned error can't find the container with id 8eca9237ef2e72f8839e1d94ae7f7bed67f833431a8b64cc60fb4ce53d8acfe0
Oct 30 23:18:56.476019 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.475981    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:56.476190 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.476043    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:56.476190 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:56.476142    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:18:56.476271 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:56.476206    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:57.476186661 +0000 UTC m=+37.575208901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : configmap "kube-rbac-proxy" not found
Oct 30 23:18:56.476699 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:56.476608    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:18:56.476699 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:56.476661    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:57.47664702 +0000 UTC m=+37.575669260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:18:56.638870 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.638822    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vlbq7-jsbgj" event={"ID":"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06","Type":"ContainerStarted","Data":"8eca9237ef2e72f8839e1d94ae7f7bed67f833431a8b64cc60fb4ce53d8acfe0"}
Oct 30 23:18:56.744498 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.744411    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rhc9c"]
Oct 30 23:18:56.767502 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.767459    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rhc9c"]
Oct 30 23:18:56.767662 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.767587    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:56.776312 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.776259    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 23:18:56.776453 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.776321    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 23:18:56.776453 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.776322    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 23:18:56.776453 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.776355    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 23:18:56.776453 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.776443    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8hnlh\""
Oct 30 23:18:56.854970 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.854770    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz"]
Oct 30 23:18:56.878383 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.878336    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz"]
Oct 30 23:18:56.878533 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.878479    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz"
Oct 30 23:18:56.879019 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.878993    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lstnh\" (UniqueName: \"kubernetes.io/projected/8919fd44-7b88-4215-9d80-50cf73eb1de7-kube-api-access-lstnh\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:56.879155 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.879043    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-key\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:56.879155 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.879063    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-cabundle\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:56.884599 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.884563    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qrvbx\""
Oct 30 23:18:56.884599 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.884586    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 23:18:56.885074 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.885037    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 23:18:56.979706 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.979667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lstnh\" (UniqueName: \"kubernetes.io/projected/8919fd44-7b88-4215-9d80-50cf73eb1de7-kube-api-access-lstnh\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:56.979858 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.979724    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t869h\" (UniqueName: \"kubernetes.io/projected/78e7226a-8a87-4a93-8c7b-642ff889d7d0-kube-api-access-t869h\") pod \"migrator-66df5d496b-58gzz\" (UID: \"78e7226a-8a87-4a93-8c7b-642ff889d7d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz"
Oct 30 23:18:56.979858 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.979762    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-key\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:56.979858 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.979788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-cabundle\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:56.980577 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.980551    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-cabundle\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:56.983782 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.983759    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-key\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:56.997459 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:56.997394    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lstnh\" (UniqueName: \"kubernetes.io/projected/8919fd44-7b88-4215-9d80-50cf73eb1de7-kube-api-access-lstnh\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:57.080775 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.080739    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t869h\" (UniqueName: \"kubernetes.io/projected/78e7226a-8a87-4a93-8c7b-642ff889d7d0-kube-api-access-t869h\") pod \"migrator-66df5d496b-58gzz\" (UID: \"78e7226a-8a87-4a93-8c7b-642ff889d7d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz"
Oct 30 23:18:57.094699 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.094671    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t869h\" (UniqueName: \"kubernetes.io/projected/78e7226a-8a87-4a93-8c7b-642ff889d7d0-kube-api-access-t869h\") pod \"migrator-66df5d496b-58gzz\" (UID: \"78e7226a-8a87-4a93-8c7b-642ff889d7d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz"
Oct 30 23:18:57.096387 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.096347    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:18:57.202749 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.202722    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz"
Oct 30 23:18:57.405963 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.405929    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rhc9c"]
Oct 30 23:18:57.422055 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:57.422022    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8919fd44_7b88_4215_9d80_50cf73eb1de7.slice/crio-e4d16b294622fdc5f772ed7c49f04dd8bc130e6004ace69b13fd9a68e0f10077 WatchSource:0}: Error finding container e4d16b294622fdc5f772ed7c49f04dd8bc130e6004ace69b13fd9a68e0f10077: Status 404 returned error can't find the container with id e4d16b294622fdc5f772ed7c49f04dd8bc130e6004ace69b13fd9a68e0f10077
Oct 30 23:18:57.493874 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.493835    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:57.494010 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.493898    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:57.494046 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:57.494022    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:18:57.494129 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:57.494089    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:59.494069882 +0000 UTC m=+39.593092125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : configmap "kube-rbac-proxy" not found
Oct 30 23:18:57.494576 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:57.494554    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:18:57.494691 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:57.494612    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:18:59.494595936 +0000 UTC m=+39.593618163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:18:57.499040 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.499018    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz"]
Oct 30 23:18:57.503441 ip-10-0-134-83 kubenswrapper[2571]: W1030 23:18:57.503410    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e7226a_8a87_4a93_8c7b_642ff889d7d0.slice/crio-be64b56d12379c6cadd3ac6366af4a842978e3d06bd2dd131536716efb948e8f WatchSource:0}: Error finding container be64b56d12379c6cadd3ac6366af4a842978e3d06bd2dd131536716efb948e8f: Status 404 returned error can't find the container with id be64b56d12379c6cadd3ac6366af4a842978e3d06bd2dd131536716efb948e8f
Oct 30 23:18:57.643713 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.643628    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c" event={"ID":"8919fd44-7b88-4215-9d80-50cf73eb1de7","Type":"ContainerStarted","Data":"e4d16b294622fdc5f772ed7c49f04dd8bc130e6004ace69b13fd9a68e0f10077"}
Oct 30 23:18:57.645385 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.645312    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz" event={"ID":"78e7226a-8a87-4a93-8c7b-642ff889d7d0","Type":"ContainerStarted","Data":"be64b56d12379c6cadd3ac6366af4a842978e3d06bd2dd131536716efb948e8f"}
Oct 30 23:18:57.999483 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:57.999404    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:18:57.999699 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:57.999595    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:18:57.999699 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:57.999661    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls podName:82ca7641-8711-4a48-86e0-a59f3ca8e498 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:05.999642739 +0000 UTC m=+46.098664968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls") pod "dns-default-b4l22" (UID: "82ca7641-8711-4a48-86e0-a59f3ca8e498") : secret "dns-default-metrics-tls" not found
Oct 30 23:18:58.650775 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:58.650737    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzvtj" event={"ID":"f223a1d7-7a59-45a5-970f-deecf721ff88","Type":"ContainerStarted","Data":"2eba7e0d8c5184c91b0714a001302425868edd7203b319cc391bb2b340ea6451"}
Oct 30 23:18:58.651523 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:58.651504    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:18:59.512120 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:59.512085    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:59.512264 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:59.512215    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:18:59.512327 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:59.512280    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:18:59.512327 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:59.512321    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:18:59.512459 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:59.512358    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:03.512340857 +0000 UTC m=+43.611363289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : configmap "kube-rbac-proxy" not found
Oct 30 23:18:59.512459 ip-10-0-134-83 kubenswrapper[2571]: E1030 23:18:59.512400    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:03.512386275 +0000 UTC m=+43.611408500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:18:59.656233 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:59.655971    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vlbq7-jsbgj" event={"ID":"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06","Type":"ContainerStarted","Data":"d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e"}
Oct 30 23:18:59.675921 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:59.675871    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vlbq7-jsbgj" podStartSLOduration=1.637322079 podStartE2EDuration="4.675856833s" totalImagesPullingTime="3.038534754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:59.673452771 +0000 UTC m=+39.772475019" watchObservedRunningTime="2026-10-30 23:18:59.675856833 +0000 UTC m=+39.774879082"
Oct 30 23:18:59.676090 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:18:59.675984    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vzvtj" podStartSLOduration=34.908456983 podStartE2EDuration="38.675975919s" totalImagesPullingTime="3.767518936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:58.668868121 +0000 UTC m=+38.767890370" watchObservedRunningTime="2026-10-30 23:18:59.675975919 +0000 UTC m=+39.774998166"
Oct 30 23:19:00.660324 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:19:00.660144    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz" event={"ID":"78e7226a-8a87-4a93-8c7b-642ff889d7d0","Type":"ContainerStarted","Data":"868070e45b7b19d9767da36a21da4dd771f7af553c0c24af86f1854d31071a85"}
Oct 30 23:19:00.678207 ip-10-0-134-83 kubenswrapper[2571]: I1030 23:19:00.678158    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz" podStartSLOduration=2.243565695 podStartE2EDuration="4.678143487s" totalImagesPullingTime="2.434577792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:00.677208393 +0000 UTC m=+40.776230644" watchObservedRunningTime="2026-10-30 23:19:00.678143487 +0000 UTC m=+40.777165734"
Oct 30 23:19:00.995215 ip-10-0-134-83 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 23:19:01.008063 ip-10-0-134-83 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 23:19:01.008314 ip-10-0-134-83 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 23:19:01.008401 ip-10-0-134-83 systemd[1]: kubelet.service: Consumed 2.494s CPU time.
Oct 30 23:19:01.022998 ip-10-0-134-83 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 23:19:01.121165 ip-10-0-134-83 kubenswrapper[4555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:19:01.121165 ip-10-0-134-83 kubenswrapper[4555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 23:19:01.121165 ip-10-0-134-83 kubenswrapper[4555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:19:01.121165 ip-10-0-134-83 kubenswrapper[4555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:19:01.123501 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123412    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:19:01.123501 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123501    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:19:01.123501 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123505    4555 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123510    4555 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123513    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123517    4555 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123520    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123522    4555 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123525    4555 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123528    4555 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123530    4555 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123533    4555 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123536    4555 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123538    4555 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123541    4555 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123545    4555 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123549    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123552    4555 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123556    4555 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123558    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123561    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:19:01.123608 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123564    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123566    4555 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123569    4555 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123572    4555 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123575    4555 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123585    4555 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123588    4555 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123591    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123593    4555 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123596    4555 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123599    4555 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123601    4555 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123604    4555 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123607    4555 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123610    4555 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123612    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123615    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123618    4555 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123621    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123624    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:19:01.124152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123626    4555 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123629    4555 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123631    4555 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123634    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123636    4555 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123639    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123642    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123644    4555 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123648    4555 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123651    4555 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123654    4555 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123657    4555 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123660    4555 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123662    4555 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123665    4555 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123668    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123670    4555 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123673    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123675    4555 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:19:01.124721 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123678    4555 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123681    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123683    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123686    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123688    4555 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123692    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123694    4555 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123697    4555 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123699    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123702    4555 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123705    4555 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123707    4555 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123716    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123719    4555 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123721    4555 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123724    4555 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123727    4555 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123731    4555 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123735    4555 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:19:01.125253 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123738    4555 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123741    4555 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123744    4555 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123747    4555 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123750    4555 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123753    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123755    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123758    4555 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123761    4555 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123763    4555 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123766    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123768    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123771    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123774    4555 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123776    4555 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123779    4555 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123781    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123784    4555 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123787    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:19:01.125779 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123789    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123792    4555 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123796    4555 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123798    4555 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123801    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123803    4555 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123806    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.123814    4555 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124604    4555 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124612    4555 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124615    4555 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124617    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124620    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124623    4555 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124625    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124628    4555 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124631    4555 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124634    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124637    4555 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124639    4555 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:19:01.126295 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124642    4555 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124645    4555 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124648    4555 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124650    4555 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124653    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124655    4555 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124658    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124661    4555 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124663    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124666    4555 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124668    4555 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124671    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124675    4555 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124677    4555 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124680    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124683    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124685    4555 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124688    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124691    4555 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124693    4555 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:19:01.126861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124696    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124701    4555 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124706    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124709    4555 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124712    4555 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124715    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124717    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124720    4555 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124723    4555 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124726    4555 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124729    4555 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124732    4555 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124735    4555 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124737    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124740    4555 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124742    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124745    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124748    4555 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:19:01.127416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124751    4555 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124754    4555 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124757    4555 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124759    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124762    4555 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124765    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124767    4555 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124770    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124773    4555 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124775    4555 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124778    4555 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124780    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124783    4555 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124786    4555 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124789    4555 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124798    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124801    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124804    4555 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124808    4555 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:19:01.127922 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124812    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124815    4555 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124818    4555 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124820    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124823    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124825    4555 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124828    4555 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124831    4555 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124833    4555 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124836    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124838    4555 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124841    4555 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124844    4555 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124847    4555 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124850    4555 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124852    4555 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124855    4555 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124858    4555 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124860    4555 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124863    4555 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:19:01.128451 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124866    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124868    4555 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124872    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124874    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124877    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124879    4555 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124882    4555 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124893    4555 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124896    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124900    4555 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124903    4555 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124906    4555 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124908    4555 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124916    4555 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124918    4555 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124921    4555 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.124923    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125423    4555 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125428    4555 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125431    4555 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:19:01.128984 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125434    4555 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125437    4555 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125440    4555 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125443    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125446    4555 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125448    4555 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125451    4555 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125453    4555 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125457    4555 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125459    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125462    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125464    4555 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125467    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125469    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125472    4555 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125475    4555 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125478    4555 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125480    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125483    4555 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125486    4555 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:19:01.129566 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125488    4555 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125491    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125494    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125497    4555 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125499    4555 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125502    4555 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125504    4555 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125507    4555 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125509    4555 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125512    4555 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125515    4555 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125520    4555 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125523    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125526    4555 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125529    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125532    4555 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125535    4555 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125537    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125540    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:19:01.130164 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125543    4555 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125545    4555 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125548    4555 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125550    4555 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125553    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125557    4555 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125561    4555 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125564    4555 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125566    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125569    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125572    4555 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125575    4555 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125577    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125580    4555 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125582    4555 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125586    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125588    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125592    4555 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125594    4555 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:19:01.130696 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125597    4555 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125600    4555 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125602    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125605    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125607    4555 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125610    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125613    4555 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125615    4555 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125618    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125620    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125623    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125625    4555 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125628    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125631    4555 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125633    4555 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125636    4555 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125639    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125641    4555 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125644    4555 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:19:01.131201 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125646    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125649    4555 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125651    4555 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125654    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125657    4555 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125659    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125662    4555 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125664    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125667    4555 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125674    4555 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125677    4555 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125681    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125684    4555 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125686    4555 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125689    4555 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125692    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125694    4555 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125696    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125699    4555 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125702    4555 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:19:01.131712 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125704    4555 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125707    4555 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125709    4555 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125712    4555 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125714    4555 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.125717    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125815    4555 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125822    4555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125829    4555 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125833    4555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125838    4555 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125841    4555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125846    4555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125850    4555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125854    4555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125857    4555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125860    4555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125864    4555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125867    4555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125870    4555 flags.go:64] FLAG: --cgroup-root=""
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125873    4555 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125878    4555 flags.go:64] FLAG: --client-ca-file=""
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125881    4555 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 23:19:01.132265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125884    4555 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125889    4555 flags.go:64] FLAG: --cluster-domain=""
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125892    4555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125895    4555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125898    4555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125902    4555 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125906    4555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125909    4555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125912    4555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125915    4555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125919    4555 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125922    4555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125924    4555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125928    4555 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125931    4555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125935    4555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125938    4555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125941    4555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125944    4555 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125947    4555 flags.go:64] FLAG: --enable-server="true"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125950    4555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125954    4555 flags.go:64] FLAG: --event-burst="100"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125957    4555 flags.go:64] FLAG: --event-qps="50"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125960    4555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 23:19:01.133007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125964    4555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125967    4555 flags.go:64] FLAG: --eviction-hard=""
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125971    4555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125975    4555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125978    4555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125981    4555 flags.go:64] FLAG: --eviction-soft=""
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125986    4555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125989    4555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125992    4555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125995    4555 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.125998    4555 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126001    4555 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126004    4555 flags.go:64] FLAG: --feature-gates=""
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126008    4555 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126012    4555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126015    4555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126018    4555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126022    4555 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126025    4555 flags.go:64] FLAG: --help="false"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126028    4555 flags.go:64] FLAG: --hostname-override="ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126032    4555 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126036    4555 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126039    4555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 23:19:01.133767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126043    4555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126046    4555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126049    4555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126052    4555 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126055    4555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126058    4555 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126061    4555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126064    4555 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126067    4555 flags.go:64] FLAG: --kube-reserved=""
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126070    4555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126073    4555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126076    4555 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126079    4555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126082    4555 flags.go:64] FLAG: --lock-file=""
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126085    4555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126089    4555 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126092    4555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126097    4555 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126100    4555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126104    4555 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126107    4555 flags.go:64] FLAG: --logging-format="text"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126110    4555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126114    4555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126116    4555 flags.go:64] FLAG: --manifest-url=""
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126119    4555 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 23:19:01.134430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126124    4555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126127    4555 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126130    4555 flags.go:64] FLAG: --max-pods="110"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126133    4555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126137    4555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126140    4555 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126143    4555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126146    4555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126150    4555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126153    4555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126160    4555 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126163    4555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126166    4555 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126169    4555 flags.go:64] FLAG: --pod-cidr=""
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126172    4555 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126175    4555 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126178    4555 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126181    4555 flags.go:64] FLAG: --port="10250"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126184    4555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126187    4555 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-064b3e15d49d28a10"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126190    4555 flags.go:64] FLAG: --qos-reserved=""
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126193    4555 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126197    4555 flags.go:64] FLAG: --register-node="true"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126200    4555 flags.go:64] FLAG: --register-with-taints=""
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126204    4555 flags.go:64] FLAG: --registry-burst="10"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126207    4555 flags.go:64] FLAG: --registry-qps="5"
Oct 30 23:19:01.135480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126210    4555 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126214    4555 flags.go:64] FLAG: --reserved-memory=""
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126217    4555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126221    4555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126224    4555 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126227    4555 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126230    4555 flags.go:64] FLAG: --runonce="false"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126232    4555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126236    4555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126239    4555 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126243    4555 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126246    4555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126249    4555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126253    4555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126256    4555 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126259    4555 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126262    4555 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126265    4555 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126267    4555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126271    4555 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126274    4555 flags.go:64] FLAG: --system-cgroups=""
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126277    4555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126282    4555 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126285    4555 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126288    4555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 23:19:01.136455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126292    4555 flags.go:64] FLAG: --tls-min-version=""
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126295    4555 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126298    4555 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126303    4555 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126306    4555 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126309    4555 flags.go:64] FLAG: --v="2"
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126313    4555 flags.go:64] FLAG: --version="false"
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126317    4555 flags.go:64] FLAG: --vmodule=""
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126321    4555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126325    4555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126498    4555 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126503    4555 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126507    4555 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126510    4555 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126512    4555 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126516    4555 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126518    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126521    4555 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126524    4555 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126527    4555 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126530    4555 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126532    4555 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:19:01.137147 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126535    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126537    4555 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126540    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126543    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126545    4555 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126548    4555 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126550    4555 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126553    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126556    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126559    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126561    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126564    4555 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126566    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126570    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126573    4555 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126576    4555 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126578    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126581    4555 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126583    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:19:01.137874 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126586    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126594    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126597    4555 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126600    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126603    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126605    4555 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126608    4555 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126610    4555 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126613    4555 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126616    4555 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126618    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126621    4555 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126624    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126626    4555 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126629    4555 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126632    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126635    4555 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126637    4555 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126640    4555 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126642    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:19:01.138492 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126645    4555 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126648    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126650    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126653    4555 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126655    4555 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126658    4555 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126662    4555 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126665    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126668    4555 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126671    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126675    4555 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126678    4555 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126682    4555 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126685    4555 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126693    4555 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126696    4555 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126699    4555 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126702    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126705    4555 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:19:01.139139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126708    4555 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126710    4555 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126713    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126716    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126718    4555 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126721    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126724    4555 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126726    4555 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126729    4555 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126732    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126734    4555 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126737    4555 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126740    4555 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126744    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126747    4555 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126749    4555 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126753    4555 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126755    4555 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126758    4555 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126762    4555 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:19:01.139681 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126764    4555 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126767    4555 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126769    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126772    4555 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126775    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126777    4555 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126780    4555 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126782    4555 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126790    4555 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126793    4555 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126796    4555 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126799    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126801    4555 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126804    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126807    4555 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126809    4555 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:19:01.140269 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.126837    4555 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126885    4555 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126889    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126892    4555 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126895    4555 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126897    4555 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126900    4555 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126903    4555 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126906    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126908    4555 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126911    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126913    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126917    4555 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126920    4555 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126922    4555 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126925    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126928    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126931    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126934    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:19:01.140898 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126936    4555 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126939    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126941    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126944    4555 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126947    4555 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126949    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126957    4555 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126960    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126962    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126965    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126968    4555 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126970    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126973    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126975    4555 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126978    4555 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126980    4555 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126983    4555 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126985    4555 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126988    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:19:01.141634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126990    4555 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126993    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126995    4555 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.126998    4555 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127000    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127003    4555 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127007    4555 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127009    4555 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127012    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127015    4555 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127017    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127020    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127023    4555 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127025    4555 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127028    4555 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127030    4555 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127033    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127043    4555 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127045    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:19:01.142151 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127049    4555 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127059    4555 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127062    4555 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127065    4555 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127067    4555 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127070    4555 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127073    4555 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127075    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127078    4555 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127080    4555 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127083    4555 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127086    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127088    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127091    4555 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127094    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127096    4555 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127099    4555 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127101    4555 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127104    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:19:01.142767 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127108    4555 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127110    4555 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127114    4555 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127117    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127120    4555 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127123    4555 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127126    4555 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127129    4555 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127132    4555 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127135    4555 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127137    4555 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127140    4555 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127142    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127145    4555 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127147    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127155    4555 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127158    4555 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127160    4555 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127163    4555 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127166    4555 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:19:01.143276 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127168    4555 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127171    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127173    4555 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127176    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127178    4555 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127181    4555 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127183    4555 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127186    4555 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127188    4555 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127191    4555 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.127193    4555 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.131316    4555 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.131331    4555 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.131349    4555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.131353    4555 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131443    4555 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131448    4555 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131452    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131454    4555 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:19:01.143932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131458    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131461    4555 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131464    4555 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131467    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131470    4555 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131473    4555 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131476    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131478    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131481    4555 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131483    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131486    4555 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131489    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131491    4555 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131494    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131496    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131499    4555 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131501    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131504    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131507    4555 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:19:01.144606 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131510    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131512    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131515    4555 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131517    4555 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131520    4555 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131523    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131526    4555 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131529    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131532    4555 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131535    4555 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131538    4555 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131541    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131543    4555 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131545    4555 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131548    4555 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131551    4555 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131553    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131556    4555 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131558    4555 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131561    4555 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:19:01.145148 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131563    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131566    4555 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131568    4555 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131571    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131574    4555 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131576    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131579    4555 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131596    4555 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131601    4555 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131604    4555 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131608    4555 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131611    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131614    4555 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131616    4555 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131619    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131622    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131624    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131627    4555 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131630    4555 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131633    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:19:01.145698 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131635    4555 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131638    4555 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131641    4555 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131644    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131646    4555 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131649    4555 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131651    4555 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131654    4555 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131656    4555 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131659    4555 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131662    4555 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131666    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131670    4555 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131673    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131676    4555 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131679    4555 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131682    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131684    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131687    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:19:01.146235 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131689    4555 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131693    4555 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131696    4555 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131698    4555 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131701    4555 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131703    4555 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131707    4555 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131711    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131714    4555 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131718    4555 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131721    4555 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131723    4555 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131726    4555 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131729    4555 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131732    4555 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131735    4555 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131737    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131741    4555 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131743    4555 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:19:01.146758 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131746    4555 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131748    4555 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131751    4555 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131753    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131756    4555 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.131784    4555 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131970    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131979    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131984    4555 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131988    4555 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131993    4555 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.131997    4555 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132000    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132004    4555 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132009    4555 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132013    4555 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:19:01.147281 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132017    4555 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132021    4555 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132025    4555 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132029    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132033    4555 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132037    4555 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132041    4555 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132045    4555 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132049    4555 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132053    4555 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132057    4555 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132062    4555 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132066    4555 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132070    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132074    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132077    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132082    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132085    4555 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132089    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132094    4555 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:19:01.147722 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132098    4555 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132102    4555 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132106    4555 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132109    4555 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132113    4555 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132118    4555 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132122    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132125    4555 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132129    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132132    4555 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132136    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132140    4555 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132144    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132148    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132152    4555 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132155    4555 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132159    4555 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132163    4555 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132167    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:19:01.148258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132170    4555 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132174    4555 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132178    4555 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132182    4555 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132186    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132191    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132196    4555 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132200    4555 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132204    4555 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132208    4555 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132212    4555 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132217    4555 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132221    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132225    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132229    4555 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132233    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132240    4555 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132245    4555 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132248    4555 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:19:01.148777 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132251    4555 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132254    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132257    4555 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132259    4555 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132262    4555 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132265    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132267    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132270    4555 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132274    4555 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132276    4555 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132279    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132282    4555 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132285    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132287    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132290    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132292    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132295    4555 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132297    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132300    4555 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:19:01.149287 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132303    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132306    4555 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132309    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132311    4555 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132317    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132319    4555 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132321    4555 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132328    4555 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132330    4555 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132333    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132338    4555 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132342    4555 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132346    4555 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132349    4555 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132351    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132354    4555 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132356    4555 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132359    4555 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:19:01.149886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132362    4555 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.132425    4555 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132464    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132469    4555 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132472    4555 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132475    4555 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132478    4555 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132481    4555 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132485    4555 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132487    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132490    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132493    4555 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132495    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132499    4555 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132503    4555 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:19:01.150494 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132512    4555 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132515    4555 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132518    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132522    4555 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132525    4555 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132528    4555 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132531    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132535    4555 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132538    4555 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132540    4555 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132543    4555 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132545    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132548    4555 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132551    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132554    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132557    4555 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132559    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132562    4555 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132564    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:19:01.151166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132567    4555 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132570    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132572    4555 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132574    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132577    4555 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132580    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132582    4555 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132585    4555 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132587    4555 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132590    4555 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132593    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132597    4555 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132600    4555 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132603    4555 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132607    4555 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132610    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132613    4555 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132616    4555 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132619    4555 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:19:01.152025 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132622    4555 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132625    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132628    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132631    4555 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132633    4555 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132636    4555 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132638    4555 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132641    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132643    4555 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132647    4555 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132650    4555 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132652    4555 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132655    4555 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132657    4555 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132660    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132662    4555 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132665    4555 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132667    4555 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132670    4555 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132672    4555 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:19:01.152841 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132675    4555 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132677    4555 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132680    4555 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132683    4555 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132685    4555 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132688    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132690    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132693    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132696    4555 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132700    4555 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132702    4555 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132705    4555 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132707    4555 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132711    4555 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132714    4555 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132716    4555 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132719    4555 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132721    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132724    4555 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132726    4555 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:19:01.153644 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132729    4555 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132731    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132734    4555 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132736    4555 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132739    4555 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132741    4555 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132744    4555 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132747    4555 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132749    4555 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132752    4555 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132754    4555 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132757    4555 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132759    4555 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132762    4555 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:01.132764    4555 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.132821    4555 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.133806    4555 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.133858    4555 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.134047    4555 server.go:1014] "Starting client certificate rotation"
Oct 30 23:19:01.154166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.134252    4555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:19:01.154846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.134302    4555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:13:20 +0000 UTC" deadline="2028-05-16 03:14:16.058211263 +0000 UTC"
Oct 30 23:19:01.154846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.134388    4555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13515h55m14.923827041s"
Oct 30 23:19:01.154846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.134738    4555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:19:01.154846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.137045    4555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:19:01.154846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.138510    4555 log.go:25] "Validated CRI v1 runtime API"
Oct 30 23:19:01.154846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.140219    4555 log.go:25] "Validated CRI v1 image API"
Oct 30 23:19:01.154846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.141040    4555 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 23:19:01.154846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.142681    4555 fs.go:135] Filesystem UUIDs: map[7059bbec-926b-43d2-9f92-19a42ac5ce25:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cad86b51-2eac-488c-adc0-7c712f6bb863:/dev/nvme0n1p4]
Oct 30 23:19:01.155088 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.142701    4555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b2edeb6a09fc60695979195b8fdc000a2525c795ff20fd8342e7421e5d9568d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b2edeb6a09fc60695979195b8fdc000a2525c795ff20fd8342e7421e5d9568d/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2536bb12e29523bf461b1335af480a645e70794c776fdc06e8c40670ddba4810/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2536bb12e29523bf461b1335af480a645e70794c776fdc06e8c40670ddba4810/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3773ebbbf252b2ecc7853e5584eccf71a5caabac272710021c931fcc1dc77ee7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3773ebbbf252b2ecc7853e5584eccf71a5caabac272710021c931fcc1dc77ee7/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b706a7844f9e86078423190b2efcfadff1c2c63edf30fec9c714f17be65ac13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b706a7844f9e86078423190b2efcfadff1c2c63edf30fec9c714f17be65ac13/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e21826aebd0c6f3feb0242e77c2a4ec37013076e09349310115e507556f3cd6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e21826aebd0c6f3feb0242e77c2a4ec37013076e09349310115e507556f3cd6/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8eca9237ef2e72f8839e1d94ae7f7bed67f833431a8b64cc60fb4ce53d8acfe0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8eca9237ef2e72f8839e1d94ae7f7bed67f833431a8b64cc60fb4ce53d8acfe0/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a7428cb5ca52fe976082c38a46ce1d0362db76ac71c075eeeadd9c0b4bd7916/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a7428cb5ca52fe976082c38a46ce1d0362db76ac71c075eeeadd9c0b4bd7916/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0122039b3f65e344e980f1d72bcee94afa767fe506e91be5d820c40daf6e64d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0122039b3f65e344e980f1d72bcee94afa767fe506e91be5d820c40daf6e64d/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a25a3119963b4699abcb98d9a01469e04c4d900e809719ece8011ceec2ccdf93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a25a3119963b4699abcb98d9a01469e04c4d900e809719ece8011ceec2ccdf93/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be64b56d12379c6cadd3ac6366af4a842978e3d06bd2dd131536716efb948e8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be64b56d12379c6cadd3ac6366af4a842978e3d06bd2dd131536716efb948e8f/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c872e40ef133a0dec402980ced6bc6ab41f1aa50063cf9fb7d567c68b53b1693/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c872e40ef133a0dec402980ced6bc6ab41f1aa50063cf9fb7d567c68b53b1693/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2a4554309cb996246b363eb61c06dd68655660435c9d7a2efd7c7c351226321/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2a4554309cb996246b363eb61c06dd68655660435c9d7a2efd7c7c351226321/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4d16b294622fdc5f772ed7c49f04dd8bc130e6004ace69b13fd9a68e0f10077/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4d16b294622fdc5f772ed7c49f04dd8bc130e6004ace69b13fd9a68e0f10077/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4fdacd55354bb13b061bd099d0c6db4fd948027c7d7371f4445e3a428d2fd73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4fdacd55354bb13b061bd099d0c6db4fd948027c7d7371f4445e3a428d2fd73/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea556f7c8aa60bbf9382abb1db8cb12713f4b57e482ce96d73e41efbe8004481/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea556f7c8aa60bbf9382abb1db8cb12713f4b57e482ce96d73e41efbe8004481/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f98d49b9315b6f83e51dd5fff61e432771d5ec4221c2318a8d136a4ccc3f90a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f98d49b9315b6f83e51dd5fff61e432771d5ec4221c2318a8d136a4ccc3f90a5/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/093e503b-974c-4918-9a88-0de16b2aa06d/volumes/kubernetes.io~projected/kube-api-access-wr8cj:{mountpoint:/var/lib/kubelet/pods/093e503b-974c-4918-9a88-0de16b2aa06d/volumes/kubernetes.io~projected/kube-api-access-wr8cj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/321dcdee-cb90-4de4-8c18-39120efda8e4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/321dcdee-cb90-4de4-8c18-39120efda8e4/volumes/kubernetes.io~secret/agent-certs major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06/volumes/kubernetes.io~projected/kube-api-access-5mffv:{mountpoint:/var/lib/kubelet/pods/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06/volumes/kubernetes.io~projected/kube-api-access-5mffv major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54da46ee-38c8-441e-a219-552cd01c7340/volumes/kubernetes.io~projected/kube-api-access-dlfq2:{mountpoint:/var/lib/kubelet/pods/54da46ee-38c8-441e-a219-552cd01c7340/volumes/kubernetes.io~projected/kube-api-access-dlfq2 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52/volumes/kubernetes.io~projected/kube-api-access-jpkvp:{mountpoint:/var/lib/kubelet/pods/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52/volumes/kubernetes.io~projected/kube-api-access-jpkvp major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6748f5ad-4980-4e4b-b448-bdc7f50e9e57/volumes/kubernetes.io~projected/kube-api-access-k8zh6:{mountpoint:/var/lib/kubelet/pods/6748f5ad-4980-4e4b-b448-bdc7f50e9e57/volumes/kubernetes.io~projected/kube-api-access-k8zh6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f805291-3eab-499d-aed8-577d64cf5c8c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6f805291-3eab-499d-aed8-577d64cf5c8c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f805291-3eab-499d-aed8-577d64cf5c8c/volumes/kubernetes.io~projected/kube-api-access-wm86f:{mountpoint:/var/lib/kubelet/pods/6f805291-3eab-499d-aed8-577d64cf5c8c/volumes/kubernetes.io~projected/kube-api-access-wm86f major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f805291-3eab-499d-aed8-577d64cf5c8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6f805291-3eab-499d-aed8-577d64cf5c8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78e7226a-8a87-4a93-8c7b-642ff889d7d0/volumes/kubernetes.io~projected/kube-api-access-t869h:{mountpoint:/var/lib/kubelet/pods/78e7226a-8a87-4a93-8c7b-642ff889d7d0/volumes/kubernetes.io~projected/kube-api-access-t869h major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82ca7641-8711-4a48-86e0-a59f3ca8e498/volumes/kubernetes.io~projected/kube-api-access-jf4rz:{mountpoint:/var/lib/kubelet/pods/82ca7641-8711-4a48-86e0-a59f3ca8e498/volumes/kubernetes.io~projected/kube-api-access-jf4rz major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83690100-e937-4dba-80e5-b5af4b7e507d/volumes/kubernetes.io~projected/kube-api-access-86llh:{mountpoint:/var/lib/kubelet/pods/83690100-e937-4dba-80e5-b5af4b7e507d/volumes/kubernetes.io~projected/kube-api-access-86llh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8919fd44-7b88-4215-9d80-50cf73eb1de7/volumes/kubernetes.io~projected/kube-api-access-lstnh:{mountpoint:/var/lib/kubelet/pods/8919fd44-7b88-4215-9d80-50cf73eb1de7/volumes/kubernetes.io~projected/kube-api-access-lstnh major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8919fd44-7b88-4215-9d80-50cf73eb1de7/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/8919fd44-7b88-4215-9d80-50cf73eb1de7/volumes/kubernetes.io~secret/signing-key major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99889cd4-347f-46b6-938f-0886271bb04d/volumes/kubernetes.io~projected/kube-api-access-nc2ft:{mountpoint:/var/lib/kubelet/pods/99889cd4-347f-46b6-938f-0886271bb04d/volumes/kubernetes.io~projected/kube-api-access-nc2ft major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d012abb7-7079-484a-8f4c-46b9c8bed1a4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d012abb7-7079-484a-8f4c-46b9c8bed1a4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d012abb7-7079-484a-8f4c-46b9c8bed1a4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d012abb7-7079-484a-8f4c-46b9c8bed1a4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d012abb7-7079-484a-8f4c-46b9c8bed1a4/volumes/kubernetes.io~projected/kube-api-access-q9fpl:{mountpoint:/var/lib/kubelet/pods/d012abb7-7079-484a-8f4c-46b9c8bed1a4/volumes/kubernetes.io~projected/kube-api-access-q9fpl major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d/volumes/kubernetes.io~projected/kube-api-access-x55vs:{mountpoint:/var/lib/kubelet/pods/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d/volumes/kubernetes.io~projected/kube-api-access-x55vs major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f223a1d7-7a59-45a5-970f-deecf721ff88/volumes/kubernetes.io~projected/kube-api-access-pzwqv:{mountpoint:/var/lib/kubelet/pods/f223a1d7-7a59-45a5-970f-deecf721ff88/volumes/kubernetes.io~projected/kube-api-access-pzwqv major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb61c865-8cb6-46d3-a802-82c8abfb691d/volumes/kubernetes.io~projected/kube-api-access-r8rhz:{mountpoint:/var/lib/kubelet/pods/fb61c865-8cb6-46d3-a802-82c8abfb691d/volumes/kubernetes.io~projected/kube-api-access-r8rhz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fea866f2-91be-418e-8a5f-dc609e36cfd5/volumes/kubernetes.io~projected/kube-api-access-scsx6:{mountpoint:/var/lib/kubelet/pods/fea866f2-91be-418e-8a5f-dc609e36cfd5/volumes/kubernetes.io~projected/kube-api-access-scsx6 major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2e321c43d966bbbf562b2064ed671a30a041febca6a6965d0c63181377060391/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c41e4b27aad985d098bfc58f701c86a2e6dfe4a752865137f202dec5b15018f7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/fb1634d1be3b248f198fbe69f38f0ca0d09a2fb4a3fd87a79d100f81f088fc63/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/a77b955e90f19422b8e03e7355a87d4dd4eb71f0ab0d19d1c044ef7bec1702e7/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6937c40468037d44b390e4e19c5bc7b5f04cd7df90acf207e272f5ac137d54bc/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3f0e58d8c871c06175de234102a509e9c7673493952983ce5dd4fd172f254042/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0d638f6780bf890d5208abab5f123628ed9ba64a5371a82e4aa36241851b1f47/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/fe2fc862d0f519a2ac9b7e1c9816b01cb120fc1d3a02ac6a211109a753bdd247/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e959a882984e1881255d87b254a529da36c9e823bc844b424439b8cee7bb6679/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c5abe3b8c0436a59c9e74dac3a7ca94ac507760f44ac6be7424e1f6f4f9965e9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/22a5f2097e3c14c18ef36e8c84cff72aa5951256d8d022d511d254fe479ba5ba/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/90edd3089276eba156a8788a048c709209215b5c97f4f210e22981fac49beadf/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/80bd6502aaa2674ac48b16f2121d49d3790a6cc39ed43447b5f6437a90a102ac/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9f0b3ea5fa90f7449ffcff3b2d4b92dba8cb67c1f2ac39c013715380289f2338/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/8a98fe311114c102b507a955d1737e08114e0c8e71e3f8440dc397d0623bbeb7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/dd00e0c6b25f0298cfa99318059c007d5e66e0ab9feac97c9d3ed06ae18991ec/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/be89b8f49e8b81b4827415fc7b972e9e07fd40b985ea6a3a7458f9fbcdc6033a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/66ba7b95d7656e015acfc8404c1074c6decdc86a5f76f62ebdef3172b7bc1507/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e5aad4c1ee0cdaaf253ec4cab97c808539e93be8406e621da513de5f54f0fb24/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a1b1750ab70459daec3ffe8dc7a1c03c68ee3819946932fbcadbc0af03cdb0a8/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/78810cb4da284f71dd2a058b048da57203f63d24c748c24f97d03fd001aeeb55/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/b78b5bdff13e010f2260053ab55d51e3c06aa9263fb48b294f3665c3fa86cc99/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/5481f0563e0870c995d3870b8ee9e573a803811d5c9d30d612a654d816646f42/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/2f13aeaa8e5ce2e7f000455adc0b43c61da2660203c2ec1d191d690dce28ed9c/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/dd3cd7973e81329da9b1b50b60aaff5413b8aee5ed8c5f1a39860521b89cb6c6/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/56de81b43947443b22d38521424ebd5ca967e10d0d0ecd02262111d2d1c90cda/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/7436c9bcde5419902131a5b89fc85229038d62636da354682d5fdf221fef96f2/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/0c28059488beed73e7efa55486e6f0feb4b816e67e90660d028df9d8f48a9077/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/0f177b364d131594988606e832ab920c4dc36c280b005c8936774f5dbbca34b0/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/0e016f9aea994c32cfa0795d06aa5354a1d05a4792679a1bd203c0b0e7822017/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/0cab84db4b5fee2194001f9cb302dc2f268bbc33495a082aa5301435556f3b68/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/69ea40a68e42c47568cb40d30964cca4b4735b5b4105ca811d619e4443b160df/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/f10fdf7497c9e430c112dd9434c64f7a741edf3589eadc6a1b0ccdfa42aec765/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/40235b8a1c48eecc8ff271d1bec095badf9e3cbcfab44101f9458645388b7116/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ee552b1e7152a6331686358d3b2e84acf4cff2c6a5c1ed2de7162ff33b2e16e3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f3f95294b5f4f73ccf7a27e9bcfe55ee0d98b08771a84858540d495b20e187ea/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a7d1ab09f2966077dfde33bd4a56b719fc4a0ad5fb0cb7e0649740912a5cf262/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/fc6497bc5603f6a86fe13f8f8ae0c53434d478b705531b764a32f15a75974b41/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/4480144db2d65a27f41db7af4c4cdfdfe6d44fcac36f6fecb1a2e8a157686b5f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/b6da861632b1da86b89395ea044f429f2aaa2bc3ed7ddb8b4b02496789a9498d/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4cedc7d06b60f1d8c73d256de19dfe185bbf1c1a09f54154ae597f2f90bbcc7e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/530e1b3946894a45f24a138672e9acdc26ed46421f6f99761692a1b2039932c5/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/02d42f7afe17b9a15c6356053ff3f870ed6c207d8d7efa82e06c9eb8e89656b6/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.148955    4555 manager.go:217] Machine: {Timestamp:2026-10-30 23:19:01.148479052 +0000 UTC m=+0.075116645 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099156 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20f21b94f6479a0827079abdb090e8 SystemUUID:ec20f21b-94f6-479a-0827-079abdb090e8 BootID:ee1ef1d6-0d50-4082-a5fc-cdfa7085208d Filesystems:[{Device:/var/lib/kubelet/pods/d012abb7-7079-484a-8f4c-46b9c8bed1a4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/093e503b-974c-4918-9a88-0de16b2aa06d/volumes/kubernetes.io~projected/kube-api-access-wr8cj DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a7428cb5ca52fe976082c38a46ce1d0362db76ac71c075eeeadd9c0b4bd7916/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4d16b294622fdc5f772ed7c49f04dd8bc130e6004ace69b13fd9a68e0f10077/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/321dcdee-cb90-4de4-8c18-39120efda8e4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2a4554309cb996246b363eb61c06dd68655660435c9d7a2efd7c7c351226321/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3773ebbbf252b2ecc7853e5584eccf71a5caabac272710021c931fcc1dc77ee7/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/99889cd4-347f-46b6-938f-0886271bb04d/volumes/kubernetes.io~projected/kube-api-access-nc2ft DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52/volumes/kubernetes.io~projected/kube-api-access-jpkvp DeviceMajor:0 DeviceMinor:223 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c872e40ef133a0dec402980ced6bc6ab41f1aa50063cf9fb7d567c68b53b1693/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d012abb7-7079-484a-8f4c-46b9c8bed1a4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6748f5ad-4980-4e4b-b448-bdc7f50e9e57/volumes/kubernetes.io~projected/kube-api-access-k8zh6 DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2536bb12e29523bf461b1335af480a645e70794c776fdc06e8c40670ddba4810/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/78e7226a-8a87-4a93-8c7b-642ff889d7d0/volumes/kubernetes.io~projected/kube-api-access-t869h DeviceMajor:0 DeviceMinor:230 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d012abb7-7079-484a-8f4c-46b9c8bed1a4/volumes/kubernetes.io~projected/kube-api-access-q9fpl DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6f805291-3eab-499d-aed8-577d64cf5c8c/volumes/kubernetes.io~projected/kube-api-access-wm86f DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea556f7c8aa60bbf9382abb1db8cb12713f4b57e482ce96d73e41efbe8004481/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8eca9237ef2e72f8839e1d94ae7f7bed67f833431a8b64cc60fb4ce53d8acfe0/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6f805291-3eab-499d-aed8-577d64cf5c8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d/volumes/kubernetes.io~projected/kube-api-access-x55vs DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f98d49b9315b6f83e51dd5fff61e432771d5ec4221c2318a8d136a4ccc3f90a5/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/83690100-e937-4dba-80e5-b5af4b7e507d/volumes/kubernetes.io~projected/kube-api-access-86llh DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b2edeb6a09fc60695979195b8fdc000a2525c795ff20fd8342e7421e5d9568d/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8919fd44-7b88-4215-9d80-50cf73eb1de7/volumes/kubernetes.io~projected/kube-api-access-lstnh DeviceMajor:0 DeviceMinor:229 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb61c865-8cb6-46d3-a802-82c8abfb691d/volumes/kubernetes.io~projected/kube-api-access-r8rhz DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fea866f2-91be-418e-8a5f-dc609e36cfd5/volumes/kubernetes.io~projected/kube-api-access-scsx6 DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b706a7844f9e86078423190b2efcfadff1c2c63edf30fec9c714f17be65ac13/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82ca7641-8711-4a48-86e0-a59f3ca8e498/volumes/kubernetes.io~projected/kube-api-access-jf4rz DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f223a1d7-7a59-45a5-970f-deecf721ff88/volumes/kubernetes.io~projected/kube-api-access-pzwqv DeviceMajor:0 DeviceMinor:209 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e21826aebd0c6f3feb0242e77c2a4ec37013076e09349310115e507556f3cd6/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a25a3119963b4699abcb98d9a01469e04c4d900e809719ece8011ceec2ccdf93/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0122039b3f65e344e980f1d72bcee94afa767fe506e91be5d820c40daf6e64d/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be64b56d12379c6cadd3ac6366af4a842978e3d06bd2dd131536716efb948e8f/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54da46ee-38c8-441e-a219-552cd01c7340/volumes/kubernetes.io~projected/kube-api-access-dlfq2 DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f805291-3eab-499d-aed8-577d64cf5c8c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06/volumes/kubernetes.io~projected/kube-api-access-5mffv DeviceMajor:0 DeviceMinor:222 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8919fd44-7b88-4215-9d80-50cf73eb1de7/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:228 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4fdacd55354bb13b061bd099d0c6db4fd948027c7d7371f4445e3a428d2fd73/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:8eca9237ef2e72f MacAddress:16:98:59:be:5f:18 Speed:10000 Mtu:8901} {Name:a0122039b3f65e3 MacAddress:76:45:49:1e:89:09 Speed:10000 Mtu:8901} {Name:be64b56d12379c6 MacAddress:de:a1:3a:29:2f:cb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c0:86:86:e3 Speed:0 Mtu:9001} {Name:br-int MacAddress:7a:e6:87:89:7d:67 Speed:0 Mtu:8901} {Name:d2a4554309cb996 MacAddress:ea:a3:86:22:0c:8a Speed:10000 Mtu:8901} {Name:e4d16b294622fdc MacAddress:ca:26:a3:10:ca:43 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c0:86:86:e3 Speed:0 Mtu:9001} {Name:f98d49b9315b6f8 MacAddress:d6:0d:d1:bf:ce:01 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:aa:cb:a8:e5:c0:79 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:de:d1:ec:d5:c1:ce Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.149215    4555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.149257    4555 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.149574    4555 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.149596    4555 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-83.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.149733    4555 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.149741    4555 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.149751    4555 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.149770    4555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.149864    4555 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.150025    4555 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.150124    4555 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.150157    4555 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.150166    4555 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.150175    4555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.150184    4555 kubelet.go:405] "Adding apiserver pod source"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.150194    4555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.150962    4555 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.150979    4555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:19:01.155233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.151724    4555 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 23:19:01.155897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155284    4555 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 23:19:01.155897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155302    4555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:19:01.155897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155464    4555 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 23:19:01.155897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155731    4555 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 23:19:01.155897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155774    4555 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 23:19:01.155897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155891    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155906    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155913    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155918    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155924    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155930    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155936    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155942    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155949    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155955    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155963    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155973    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.155999    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 23:19:01.156082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.156005    4555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 23:19:01.159554 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.159429    4555 server.go:1263] "Started kubelet"
Oct 30 23:19:01.159554 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.159501    4555 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 23:19:01.159835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.159778    4555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 23:19:01.159835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.159850    4555 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 23:19:01.159959 ip-10-0-134-83 systemd[1]: Started Kubernetes Kubelet.
Oct 30 23:19:01.161111 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.161014    4555 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 23:19:01.162199 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.162180    4555 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 23:19:01.162199 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.162198    4555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 23:19:01.162942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.162264    4555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:13:22 +0000 UTC" deadline="2028-03-30 01:13:15.365599821 +0000 UTC"
Oct 30 23:19:01.162942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.162294    4555 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 23:19:01.162942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.162303    4555 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 23:19:01.162942 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:01.162310    4555 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-east-2.compute.internal\" not found"
Oct 30 23:19:01.162942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.162293    4555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12385h54m14.203308865s"
Oct 30 23:19:01.162942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.162439    4555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 23:19:01.163885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163848    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" volumeName="kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 23:19:01.163990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163887    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" volumeName="kubernetes.io/empty-dir/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-archives-path" seLinuxMountContext=""
Oct 30 23:19:01.163990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163901    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8919fd44-7b88-4215-9d80-50cf73eb1de7" volumeName="kubernetes.io/secret/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-key" seLinuxMountContext=""
Oct 30 23:19:01.163990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163912    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d012abb7-7079-484a-8f4c-46b9c8bed1a4" volumeName="kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-tmp" seLinuxMountContext=""
Oct 30 23:19:01.163990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163924    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8fa85d1-6827-43dd-a947-71d6ebc1ce6d" volumeName="kubernetes.io/secret/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-cert" seLinuxMountContext=""
Oct 30 23:19:01.163990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163935    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fea866f2-91be-418e-8a5f-dc609e36cfd5" volumeName="kubernetes.io/empty-dir/fea866f2-91be-418e-8a5f-dc609e36cfd5-tmp-dir" seLinuxMountContext=""
Oct 30 23:19:01.163990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163946    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f805291-3eab-499d-aed8-577d64cf5c8c" volumeName="kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-config" seLinuxMountContext=""
Oct 30 23:19:01.163990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163958    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f805291-3eab-499d-aed8-577d64cf5c8c" volumeName="kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-env-overrides" seLinuxMountContext=""
Oct 30 23:19:01.163990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163972    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f805291-3eab-499d-aed8-577d64cf5c8c" volumeName="kubernetes.io/secret/6f805291-3eab-499d-aed8-577d64cf5c8c-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 23:19:01.163990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163984    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78e7226a-8a87-4a93-8c7b-642ff889d7d0" volumeName="kubernetes.io/projected/78e7226a-8a87-4a93-8c7b-642ff889d7d0-kube-api-access-t869h" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.163995    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d012abb7-7079-484a-8f4c-46b9c8bed1a4" volumeName="kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-tuned" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164007    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d012abb7-7079-484a-8f4c-46b9c8bed1a4" volumeName="kubernetes.io/projected/d012abb7-7079-484a-8f4c-46b9c8bed1a4-kube-api-access-q9fpl" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164018    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb61c865-8cb6-46d3-a802-82c8abfb691d" volumeName="kubernetes.io/projected/fb61c865-8cb6-46d3-a802-82c8abfb691d-kube-api-access-r8rhz" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164039    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82ca7641-8711-4a48-86e0-a59f3ca8e498" volumeName="kubernetes.io/projected/82ca7641-8711-4a48-86e0-a59f3ca8e498-kube-api-access-jf4rz" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164053    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="321dcdee-cb90-4de4-8c18-39120efda8e4" volumeName="kubernetes.io/configmap/321dcdee-cb90-4de4-8c18-39120efda8e4-konnectivity-ca" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164066    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" volumeName="kubernetes.io/projected/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-kube-api-access-5mffv" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164079    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54da46ee-38c8-441e-a219-552cd01c7340" volumeName="kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164091    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a1dd390-1d71-4270-9a6f-d9e1c66c6e52" volumeName="kubernetes.io/empty-dir/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-data-volume" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164102    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f805291-3eab-499d-aed8-577d64cf5c8c" volumeName="kubernetes.io/projected/6f805291-3eab-499d-aed8-577d64cf5c8c-kube-api-access-wm86f" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164115    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82ca7641-8711-4a48-86e0-a59f3ca8e498" volumeName="kubernetes.io/empty-dir/82ca7641-8711-4a48-86e0-a59f3ca8e498-tmp-dir" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164127    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8919fd44-7b88-4215-9d80-50cf73eb1de7" volumeName="kubernetes.io/configmap/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-cabundle" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164141    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="093e503b-974c-4918-9a88-0de16b2aa06d" volumeName="kubernetes.io/projected/093e503b-974c-4918-9a88-0de16b2aa06d-kube-api-access-wr8cj" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164153    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54da46ee-38c8-441e-a219-552cd01c7340" volumeName="kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-binary-copy" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164165    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a1dd390-1d71-4270-9a6f-d9e1c66c6e52" volumeName="kubernetes.io/projected/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-api-access-jpkvp" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164176    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6748f5ad-4980-4e4b-b448-bdc7f50e9e57" volumeName="kubernetes.io/projected/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-kube-api-access-k8zh6" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164186    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8919fd44-7b88-4215-9d80-50cf73eb1de7" volumeName="kubernetes.io/projected/8919fd44-7b88-4215-9d80-50cf73eb1de7-kube-api-access-lstnh" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164198    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99889cd4-347f-46b6-938f-0886271bb04d" volumeName="kubernetes.io/projected/99889cd4-347f-46b6-938f-0886271bb04d-kube-api-access-nc2ft" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164211    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8fa85d1-6827-43dd-a947-71d6ebc1ce6d" volumeName="kubernetes.io/projected/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-kube-api-access-x55vs" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164228    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="321dcdee-cb90-4de4-8c18-39120efda8e4" volumeName="kubernetes.io/secret/321dcdee-cb90-4de4-8c18-39120efda8e4-agent-certs" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164241    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83690100-e937-4dba-80e5-b5af4b7e507d" volumeName="kubernetes.io/projected/83690100-e937-4dba-80e5-b5af4b7e507d-kube-api-access-86llh" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164254    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6a30dfd-0560-4e49-9890-07cb7f8c8bcd" volumeName="kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164266    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fea866f2-91be-418e-8a5f-dc609e36cfd5" volumeName="kubernetes.io/projected/fea866f2-91be-418e-8a5f-dc609e36cfd5-kube-api-access-scsx6" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164278    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" volumeName="kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-service-ca-bundle" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164290    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54da46ee-38c8-441e-a219-552cd01c7340" volumeName="kubernetes.io/projected/54da46ee-38c8-441e-a219-552cd01c7340-kube-api-access-dlfq2" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164302    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83690100-e937-4dba-80e5-b5af4b7e507d" volumeName="kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-multus-daemon-config" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164313    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f223a1d7-7a59-45a5-970f-deecf721ff88" volumeName="kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164327    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="093e503b-974c-4918-9a88-0de16b2aa06d" volumeName="kubernetes.io/configmap/093e503b-974c-4918-9a88-0de16b2aa06d-iptables-alerter-script" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164338    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f805291-3eab-499d-aed8-577d64cf5c8c" volumeName="kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164349    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54da46ee-38c8-441e-a219-552cd01c7340" volumeName="kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164364    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6748f5ad-4980-4e4b-b448-bdc7f50e9e57" volumeName="kubernetes.io/configmap/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-serviceca" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164392    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82ca7641-8711-4a48-86e0-a59f3ca8e498" volumeName="kubernetes.io/configmap/82ca7641-8711-4a48-86e0-a59f3ca8e498-config-volume" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164405    4555 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83690100-e937-4dba-80e5-b5af4b7e507d" volumeName="kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-cni-binary-copy" seLinuxMountContext=""
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164417    4555 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 23:19:01.164419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.164424    4555 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 23:19:01.169078 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.169038    4555 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 23:19:01.169901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.169881    4555 factory.go:55] Registering systemd factory
Oct 30 23:19:01.169901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.169903    4555 factory.go:223] Registration of the systemd container factory successfully
Oct 30 23:19:01.170126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.170106    4555 factory.go:153] Registering CRI-O factory
Oct 30 23:19:01.170126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.170126    4555 factory.go:223] Registration of the crio container factory successfully
Oct 30 23:19:01.170272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.170186    4555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 23:19:01.170272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.170218    4555 factory.go:103] Registering Raw factory
Oct 30 23:19:01.170272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.170236    4555 manager.go:1199] Started watching for new ooms in manager
Oct 30 23:19:01.170547 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.170534    4555 manager.go:319] Starting recovery of all containers
Oct 30 23:19:01.174256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.174234    4555 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:19:01.175720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.175577    4555 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:19:01.177488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.177466    4555 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 23:19:01.178300 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.178276    4555 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:19:01.196894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.196440    4555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 23:19:01.199271 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.199232    4555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 23:19:01.199271 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.199265    4555 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 23:19:01.199484 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.199295    4555 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 23:19:01.199484 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:01.199337    4555 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 23:19:01.203490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.203443    4555 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:19:01.218720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.218687    4555 generic.go:356] "Generic (PLEG): container finished" podID="8a12773649a9a35ac251a27d3fd9a87d" containerID="6b61e5375beb0951b541e9a7492aa816f83e596d283e080f387833221aa92244" exitCode=0
Oct 30 23:19:01.231012 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.230978    4555 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="614a8c7530fec0a54ec384cda510569cab2f97d5b9d25b9db7a76cde2633d46a" exitCode=0
Oct 30 23:19:01.231177 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.231164    4555 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="0c063b6c4e119cbd063700a9906fc1ff78b5846e5c43aaae165f35a3cf04a549" exitCode=0
Oct 30 23:19:01.231321 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.231308    4555 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="2dbf9ad29af42554fdb8623420b9516127f9ffd601b5b3251eab47e7d6cba325" exitCode=0
Oct 30 23:19:01.231425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.231406    4555 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="175e5c97682cf317cc5f1451e875e955097dbb3ff888b352d63bc692478686d2" exitCode=0
Oct 30 23:19:01.231508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.231423    4555 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="8ffa6672b6e7c2da51eaac2f0588ec4328342b3dc27560eea6a22f4ce507e4dd" exitCode=0
Oct 30 23:19:01.231508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.231436    4555 generic.go:356] "Generic (PLEG): container finished" podID="54da46ee-38c8-441e-a219-552cd01c7340" containerID="68f2eef015e849ec06c4c1f1ce36fdb66b27c56768f02967d54cd527566fce94" exitCode=0
Oct 30 23:19:01.248464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.248443    4555 manager.go:324] Recovery completed
Oct 30 23:19:01.266901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.266875    4555 cpu_manager.go:226] "Starting" policy="none"
Oct 30 23:19:01.266901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.266893    4555 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 23:19:01.267039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.266913    4555 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 23:19:01.267090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.267069    4555 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 23:19:01.267131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.267083    4555 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 23:19:01.267131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.267106    4555 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 23:19:01.267131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.267114    4555 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 23:19:01.267131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.267121    4555 policy_none.go:50] "Start"
Oct 30 23:19:01.267131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.267132    4555 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 23:19:01.267269 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.267144    4555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 23:19:01.267320 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.267308    4555 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 23:19:01.267348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.267322    4555 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 23:19:01.267348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.267331    4555 policy_none.go:44] "Start"
Oct 30 23:19:01.272578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.272533    4555 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 23:19:01.276133 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.276112    4555 manager.go:341] "Starting Device Plugin manager"
Oct 30 23:19:01.276749 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:01.276138    4555 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 23:19:01.276813 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.276752    4555 server.go:87] "Starting device plugin registration server"
Oct 30 23:19:01.277026 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.277014    4555 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 23:19:01.277060 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.277029    4555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 23:19:01.277123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.277104    4555 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 23:19:01.277239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.277207    4555 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 23:19:01.277293 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.277255    4555 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 23:19:01.277293 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.277262    4555 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 23:19:01.277405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.277313    4555 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:19:01.277238872Z","UUID":"860e3b24-db20-4f44-9bd7-9e93741f98d5","Handler":null,"Name":"","Endpoint":""}
Oct 30 23:19:01.279045 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:01.279028    4555 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 23:19:01.279140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.279047    4555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 23:19:01.279140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.279074    4555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 23:19:01.299506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.299445    4555 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"]
Oct 30 23:19:01.300176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.300134    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal" event={"ID":"8a12773649a9a35ac251a27d3fd9a87d","Type":"ContainerStarted","Data":"d4a48773e7bf24e594cecfece2cd3bd7383baee652b4184cbf968ac181b1b832"}
Oct 30 23:19:01.300301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.300180    4555 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.300301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.300184    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal" event={"ID":"8a12773649a9a35ac251a27d3fd9a87d","Type":"ContainerDied","Data":"6b61e5375beb0951b541e9a7492aa816f83e596d283e080f387833221aa92244"}
Oct 30 23:19:01.300424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.300319    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal" event={"ID":"8a12773649a9a35ac251a27d3fd9a87d","Type":"ContainerStarted","Data":"3773ebbbf252b2ecc7853e5584eccf71a5caabac272710021c931fcc1dc77ee7"}
Oct 30 23:19:01.300473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.300426    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal" event={"ID":"17ddb1b82b212964d7163812b102551c","Type":"ContainerStarted","Data":"3e9614c52dac7e4219c8782a5b26c2ec2b1f9a9393d8566219388b2b36f72509"}
Oct 30 23:19:01.300473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.300444    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal" event={"ID":"17ddb1b82b212964d7163812b102551c","Type":"ContainerStarted","Data":"c872e40ef133a0dec402980ced6bc6ab41f1aa50063cf9fb7d567c68b53b1693"}
Oct 30 23:19:01.300710 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.300602    4555 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.313925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.313899    4555 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:19:01.314074 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:01.313942    4555 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.314452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.314436    4555 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:19:01.314554 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:01.314469    4555 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.377740 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.377704    4555 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:19:01.377900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.377752    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.377900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.377784    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17ddb1b82b212964d7163812b102551c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"17ddb1b82b212964d7163812b102551c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.377900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.377800    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.379894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.379871    4555 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:19:01.379988 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.379905    4555 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:19:01.379988 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.379916    4555 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:19:01.379988 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.379973    4555 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.393139 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.393114    4555 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.393236 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.393179    4555 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.478982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.478893    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17ddb1b82b212964d7163812b102551c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"17ddb1b82b212964d7163812b102551c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.478982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.478928    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.478982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.478948    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.479176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.479009    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.479176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.479016    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17ddb1b82b212964d7163812b102551c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"17ddb1b82b212964d7163812b102551c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:01.479176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:01.479073    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a12773649a9a35ac251a27d3fd9a87d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" (UID: \"8a12773649a9a35ac251a27d3fd9a87d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:02.150720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.150685    4555 apiserver.go:52] "Watching apiserver"
Oct 30 23:19:02.158657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.158623    4555 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 23:19:02.159049 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.159032    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-prlzd","openshift-network-diagnostics/network-check-target-vzvtj","openshift-dns/node-resolver-czzsf","openshift-network-operator/iptables-alerter-c7np6","openshift-dns/dns-default-b4l22","openshift-insights/periodic-gathering-vlbq7-jsbgj","openshift-ovn-kubernetes/ovnkube-node-kptck","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc","openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal","openshift-multus/multus-9x8lb","openshift-multus/network-metrics-daemon-vb7vz","openshift-cluster-node-tuning-operator/tuned-ckpdq","openshift-service-ca/service-ca-5fffc97bdf-rhc9c","kube-system/konnectivity-agent-l94rx","openshift-multus/multus-additional-cni-plugins-mlh9x","kube-system/global-pull-secret-syncer-7g6g7","openshift-ingress-canary/ingress-canary-lgv9k","openshift-insights/insights-runtime-extractor-wk72x","kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"]
Oct 30 23:19:02.162258 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.162234    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:19:02.166282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.166257    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:02.166995 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.166977    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 23:19:02.167164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167144    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 23:19:02.167234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167174    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.167234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167195    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.167234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167046    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 23:19:02.167234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167213    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.167234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167097    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.167234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167101    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.167234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167150    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.167559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.166982    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 23:19:02.167559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167153    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.167559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167030    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 23:19:02.167559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.166982    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.167559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167150    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 23:19:02.167559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167257    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.167559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167287    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.167559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167388    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 23:19:02.167559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167440    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.167559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.167482    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.168054 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168035    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.168248 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168235    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n9xrj\""
Oct 30 23:19:02.168304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168290    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 23:19:02.168423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168407    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qwtvf\""
Oct 30 23:19:02.168479 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168466    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.168479 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168472    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.168569 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168513    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 23:19:02.168641 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168623    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.168689 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168617    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 23:19:02.168689 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168655    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:19:02.168689 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168671    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4bbvk\""
Oct 30 23:19:02.168825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168689    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.168825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168731    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jt8kr\""
Oct 30 23:19:02.168825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168757    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k8dwv\""
Oct 30 23:19:02.168825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168759    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 23:19:02.168825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168781    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 23:19:02.169277 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168862    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-djdr2\""
Oct 30 23:19:02.169277 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.168924    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kc52k\""
Oct 30 23:19:02.169500 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.169485    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-82xnp\""
Oct 30 23:19:02.169721 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.169707    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2nz9z\""
Oct 30 23:19:02.169771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.169727    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9tmfn\""
Oct 30 23:19:02.169891 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.169873    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qmkhv\""
Oct 30 23:19:02.169891 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.169884    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lwvlw\""
Oct 30 23:19:02.170071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.169886    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:19:02.170071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.169983    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.170071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.169988    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.170071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.169992    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gskv9\""
Oct 30 23:19:02.170071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.170007    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 23:19:02.170071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.170013    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 23:19:02.170071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.170045    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 23:19:02.170071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.170069    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mx9rq\""
Oct 30 23:19:02.170331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.170091    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zcrv2\""
Oct 30 23:19:02.170331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.170100    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 23:19:02.170331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.170109    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.170331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.170239    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 23:19:02.170495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.170484    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.173113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173098    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 23:19:02.173215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173189    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 23:19:02.173215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173198    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.173350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173220    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 23:19:02.173350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173274    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.173350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173285    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 23:19:02.173350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173303    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 23:19:02.173350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173309    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qrvbx\""
Oct 30 23:19:02.173536 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173353    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z9rpn\""
Oct 30 23:19:02.173536 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173445    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8hnlh\""
Oct 30 23:19:02.173536 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173446    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 23:19:02.173659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.173646    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 23:19:02.183301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183282    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-system-cni-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.183407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183311    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-cnibin\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.183407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183341    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.183511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183483    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-sys\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.183566 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183533    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-tuned\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.183620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183568    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fea866f2-91be-418e-8a5f-dc609e36cfd5-hosts-file\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:19:02.183620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183589    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-multus\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.183720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183622    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-tuned\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.183720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183637    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.183720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183636    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.183720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183681    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-var-lib-kubelet\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.183887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183735    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-cnibin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.183887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183744    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.183887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183767    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-multus-daemon-config\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.183887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183803    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-socket-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.184127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183920    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-tmp\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.184127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183954    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-ovn\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183980    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-netd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.183989    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d012abb7-7079-484a-8f4c-46b9c8bed1a4-tmp\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.184127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184029    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm86f\" (UniqueName: \"kubernetes.io/projected/6f805291-3eab-499d-aed8-577d64cf5c8c-kube-api-access-wm86f\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184048    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-multus-daemon-config\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.184127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184057    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86llh\" (UniqueName: \"kubernetes.io/projected/83690100-e937-4dba-80e5-b5af4b7e507d-kube-api-access-86llh\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.184127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184081    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-netns\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184103    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-bin\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184145    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-env-overrides\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184169    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-bin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184209    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184235    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fpl\" (UniqueName: \"kubernetes.io/projected/d012abb7-7079-484a-8f4c-46b9c8bed1a4-kube-api-access-q9fpl\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184273    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-serviceca\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184308    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-env-overrides\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184347    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f805291-3eab-499d-aed8-577d64cf5c8c-ovn-node-metrics-cert\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184431    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-netns\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184459    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-sys-fs\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184487    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-serviceca\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:19:02.184501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184493    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-lib-modules\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184517    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54da46ee-38c8-441e-a219-552cd01c7340-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184533    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-host\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184564    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f805291-3eab-499d-aed8-577d64cf5c8c-ovn-node-metrics-cert\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184572    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysconfig\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184613    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-modprobe-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184638    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-device-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184666    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184691    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184727    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-config\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184742    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-kubelet-config\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184757    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.184807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184788    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184835    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-system-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184863    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-socket-dir-parent\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184910    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-kubelet\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184944    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-config\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184951    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-systemd-units\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.184985    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-slash\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185015    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185032    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-systemd\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185052    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scsx6\" (UniqueName: \"kubernetes.io/projected/fea866f2-91be-418e-8a5f-dc609e36cfd5-kube-api-access-scsx6\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185076    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-var-lib-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185139    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-cni-binary-copy\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.185186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185176    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-k8s-cni-cncf-io\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.185571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185294    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-multus-certs\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.185571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185341    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2ft\" (UniqueName: \"kubernetes.io/projected/99889cd4-347f-46b6-938f-0886271bb04d-kube-api-access-nc2ft\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:19:02.185571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185296    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83690100-e937-4dba-80e5-b5af4b7e507d-cni-binary-copy\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.185571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185364    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-run\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.185571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185413    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zh6\" (UniqueName: \"kubernetes.io/projected/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-kube-api-access-k8zh6\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:19:02.185571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185428    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.185571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185444    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-kubelet\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.185571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185480    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfq2\" (UniqueName: \"kubernetes.io/projected/54da46ee-38c8-441e-a219-552cd01c7340-kube-api-access-dlfq2\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.185571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185530    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/321dcdee-cb90-4de4-8c18-39120efda8e4-konnectivity-ca\") pod \"konnectivity-agent-l94rx\" (UID: \"321dcdee-cb90-4de4-8c18-39120efda8e4\") " pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:19:02.185571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185555    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-kubernetes\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185583    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/093e503b-974c-4918-9a88-0de16b2aa06d-iptables-alerter-script\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185699    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/093e503b-974c-4918-9a88-0de16b2aa06d-host-slash\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185729    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-host\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185746    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/321dcdee-cb90-4de4-8c18-39120efda8e4-konnectivity-ca\") pod \"konnectivity-agent-l94rx\" (UID: \"321dcdee-cb90-4de4-8c18-39120efda8e4\") " pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185753    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-systemd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185779    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-script-lib\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185805    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-node-log\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185840    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/093e503b-974c-4918-9a88-0de16b2aa06d-iptables-alerter-script\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185855    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/321dcdee-cb90-4de4-8c18-39120efda8e4-agent-certs\") pod \"konnectivity-agent-l94rx\" (UID: \"321dcdee-cb90-4de4-8c18-39120efda8e4\") " pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185879    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-registration-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.185950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185927    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea866f2-91be-418e-8a5f-dc609e36cfd5-tmp-dir\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185968    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8cj\" (UniqueName: \"kubernetes.io/projected/093e503b-974c-4918-9a88-0de16b2aa06d-kube-api-access-wr8cj\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.185988    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f805291-3eab-499d-aed8-577d64cf5c8c-ovnkube-script-lib\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186029    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-log-socket\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186040    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea866f2-91be-418e-8a5f-dc609e36cfd5-tmp-dir\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186059    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/321dcdee-cb90-4de4-8c18-39120efda8e4-agent-certs\") pod \"konnectivity-agent-l94rx\" (UID: \"321dcdee-cb90-4de4-8c18-39120efda8e4\") " pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186144    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-hostroot\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186175    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-conf-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186207    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8rhz\" (UniqueName: \"kubernetes.io/projected/fb61c865-8cb6-46d3-a802-82c8abfb691d-kube-api-access-r8rhz\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186223    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-conf\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186240    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-etc-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186254    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-etc-kubernetes\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186268    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.186325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186285    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:19:02.186758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186328    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.186758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186397    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:19:02.186758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186430    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-dbus\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:19:02.186758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186450    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:19:02.186758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.186672    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-original-pull-secret\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:19:02.201709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.201622    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86llh\" (UniqueName: \"kubernetes.io/projected/83690100-e937-4dba-80e5-b5af4b7e507d-kube-api-access-86llh\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.201709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.201647    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8rhz\" (UniqueName: \"kubernetes.io/projected/fb61c865-8cb6-46d3-a802-82c8abfb691d-kube-api-access-r8rhz\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.201903 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.201772    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2ft\" (UniqueName: \"kubernetes.io/projected/99889cd4-347f-46b6-938f-0886271bb04d-kube-api-access-nc2ft\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:19:02.202614 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.201916    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8cj\" (UniqueName: \"kubernetes.io/projected/093e503b-974c-4918-9a88-0de16b2aa06d-kube-api-access-wr8cj\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:19:02.203136 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.202933    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scsx6\" (UniqueName: \"kubernetes.io/projected/fea866f2-91be-418e-8a5f-dc609e36cfd5-kube-api-access-scsx6\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:19:02.203136 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.202938    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzwqv\" (UniqueName: \"kubernetes.io/projected/f223a1d7-7a59-45a5-970f-deecf721ff88-kube-api-access-pzwqv\") pod \"network-check-target-vzvtj\" (UID: \"f223a1d7-7a59-45a5-970f-deecf721ff88\") " pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:19:02.203443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.203390    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fpl\" (UniqueName: \"kubernetes.io/projected/d012abb7-7079-484a-8f4c-46b9c8bed1a4-kube-api-access-q9fpl\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.206939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.206919    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm86f\" (UniqueName: \"kubernetes.io/projected/6f805291-3eab-499d-aed8-577d64cf5c8c-kube-api-access-wm86f\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.206939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.206929    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zh6\" (UniqueName: \"kubernetes.io/projected/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-kube-api-access-k8zh6\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:19:02.207036 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.206954    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfq2\" (UniqueName: \"kubernetes.io/projected/54da46ee-38c8-441e-a219-552cd01c7340-kube-api-access-dlfq2\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.245753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.245727    4555 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:02.245853 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.245836    4555 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:02.257810 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.257790    4555 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:19:02.257900 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.257826    4555 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:02.258691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.258679    4555 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:19:02.258726 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.258707    4555 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-east-2.compute.internal"
Oct 30 23:19:02.263280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.263262    4555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 23:19:02.286925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.286894    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-k8s-cni-cncf-io\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.286925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.286924    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-multus-certs\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.287148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.286940    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-run\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.287148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.286955    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.287148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287014    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-multus-certs\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.287148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287017    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-k8s-cni-cncf-io\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.287148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287070    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-run\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.287148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287086    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.287148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287118    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-kubelet\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.287148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287139    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82ca7641-8711-4a48-86e0-a59f3ca8e498-config-volume\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:02.287148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287154    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-trusted-ca-bundle\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287172    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-data-volume\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287188    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-kubernetes\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287153    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-kubelet\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287239    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-kubernetes\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287294    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-data-volume\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287310    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/093e503b-974c-4918-9a88-0de16b2aa06d-host-slash\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287339    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-host\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287357    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-systemd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287396    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-node-log\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287405    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82ca7641-8711-4a48-86e0-a59f3ca8e498-config-volume\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287413    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/093e503b-974c-4918-9a88-0de16b2aa06d-host-slash\") pod \"iptables-alerter-c7np6\" (UID: \"093e503b-974c-4918-9a88-0de16b2aa06d\") " pod="openshift-network-operator/iptables-alerter-c7np6"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287423    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-registration-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287424    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-systemd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287428    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6748f5ad-4980-4e4b-b448-bdc7f50e9e57-host\") pod \"node-ca-prlzd\" (UID: \"6748f5ad-4980-4e4b-b448-bdc7f50e9e57\") " pod="openshift-image-registry/node-ca-prlzd"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287449    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-log-socket\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287473    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-log-socket\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287487    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-node-log\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287488    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-registration-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287524    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-hostroot\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287480    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-hostroot\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287550    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-conf-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287565    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-conf\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287577    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-conf-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287581    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-etc-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287599    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-etc-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.287586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287602    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-etc-kubernetes\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287622    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-etc-kubernetes\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287492    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-trusted-ca-bundle\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287620    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287662    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-cert\") pod \"ingress-canary-lgv9k\" (UID: \"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d\") " pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287678    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287683    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x55vs\" (UniqueName: \"kubernetes.io/projected/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-kube-api-access-x55vs\") pod \"ingress-canary-lgv9k\" (UID: \"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d\") " pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287695    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-conf\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287708    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287735    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287762    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-dbus\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287786    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-system-cni-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287808    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-cnibin\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287829    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-sys\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287852    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-dbus\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287873    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-system-cni-dir\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287883    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-cert\") pod \"ingress-canary-lgv9k\" (UID: \"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d\") " pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287853    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fea866f2-91be-418e-8a5f-dc609e36cfd5-hosts-file\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287901    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54da46ee-38c8-441e-a219-552cd01c7340-cnibin\") pod \"multus-additional-cni-plugins-mlh9x\" (UID: \"54da46ee-38c8-441e-a219-552cd01c7340\") " pod="openshift-multus/multus-additional-cni-plugins-mlh9x"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287904    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fea866f2-91be-418e-8a5f-dc609e36cfd5-hosts-file\") pod \"node-resolver-czzsf\" (UID: \"fea866f2-91be-418e-8a5f-dc609e36cfd5\") " pod="openshift-dns/node-resolver-czzsf"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.287875    4555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287934    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287942    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-sys\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287948    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-multus\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.287929    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-multus\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.288039    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs podName:99889cd4-347f-46b6-938f-0886271bb04d nodeName:}" failed. No retries permitted until 2026-10-30 23:19:02.788018469 +0000 UTC m=+1.714656053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs") pod "network-metrics-daemon-vb7vz" (UID: "99889cd4-347f-46b6-938f-0886271bb04d") : secret "metrics-daemon-secret" not found
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288062    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288094    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkvp\" (UniqueName: \"kubernetes.io/projected/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-api-access-jpkvp\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288120    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-var-lib-kubelet\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288144    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-cnibin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288169    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lstnh\" (UniqueName: \"kubernetes.io/projected/8919fd44-7b88-4215-9d80-50cf73eb1de7-kube-api-access-lstnh\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288199    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-cnibin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288210    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-socket-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288210    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-var-lib-kubelet\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288236    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-ovn\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288256    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-socket-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288271    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-ovn\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288272    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-netd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288296    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-netd\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288303    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4rz\" (UniqueName: \"kubernetes.io/projected/82ca7641-8711-4a48-86e0-a59f3ca8e498-kube-api-access-jf4rz\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288327    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-service-ca-bundle\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288353    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-netns\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288412    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-bin\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288437    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-bin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288447    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-run-netns\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288459    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-cni-bin\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288466    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-netns\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288481    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-var-lib-cni-bin\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288490    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-key\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288507    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-host-run-netns\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288516    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-crio-socket\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288559    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-sys-fs\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288586    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-sys-fs\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288584    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-lib-modules\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288615    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-host\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288638    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-cabundle\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288657    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-host\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288663    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288671    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-lib-modules\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288690    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysconfig\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288750    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysconfig\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.288766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288797    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-archives-path\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288823    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288848    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t869h\" (UniqueName: \"kubernetes.io/projected/78e7226a-8a87-4a93-8c7b-642ff889d7d0-kube-api-access-t869h\") pod \"migrator-66df5d496b-58gzz\" (UID: \"78e7226a-8a87-4a93-8c7b-642ff889d7d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288880    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-key\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288899    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8919fd44-7b88-4215-9d80-50cf73eb1de7-signing-cabundle\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288901    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-archives-path\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288957    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-service-ca-bundle\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.288954    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-modprobe-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289008    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-device-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289035    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289051    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289065    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-kubelet-config\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289085    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289058    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-modprobe-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289123    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289127    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6a30dfd-0560-4e49-9890-07cb7f8c8bcd-kubelet-config\") pod \"global-pull-secret-syncer-7g6g7\" (UID: \"a6a30dfd-0560-4e49-9890-07cb7f8c8bcd\") " pod="kube-system/global-pull-secret-syncer-7g6g7"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289144    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-run-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289149    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289147    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-device-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289178    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-system-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289195    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-socket-dir-parent\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289205    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-sysctl-d\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289212    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-kubelet\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289253    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-kubelet\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289253    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289265    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-systemd-units\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289284    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-multus-socket-dir-parent\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289266    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83690100-e937-4dba-80e5-b5af4b7e507d-system-cni-dir\") pod \"multus-9x8lb\" (UID: \"83690100-e937-4dba-80e5-b5af4b7e507d\") " pod="openshift-multus/multus-9x8lb"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289297    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-slash\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289289    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-systemd-units\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289322    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ca7641-8711-4a48-86e0-a59f3ca8e498-tmp-dir\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289343    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mffv\" (UniqueName: \"kubernetes.io/projected/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-kube-api-access-5mffv\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289358    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289387    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-host-slash\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289395    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-systemd\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289413    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-var-lib-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289447    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ca7641-8711-4a48-86e0-a59f3ca8e498-tmp-dir\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289464    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb61c865-8cb6-46d3-a802-82c8abfb691d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwfdc\" (UID: \"fb61c865-8cb6-46d3-a802-82c8abfb691d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfdc"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289466    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d012abb7-7079-484a-8f4c-46b9c8bed1a4-etc-systemd\") pod \"tuned-ckpdq\" (UID: \"d012abb7-7079-484a-8f4c-46b9c8bed1a4\") " pod="openshift-cluster-node-tuning-operator/tuned-ckpdq"
Oct 30 23:19:02.290965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.289466    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f805291-3eab-499d-aed8-577d64cf5c8c-var-lib-openvswitch\") pod \"ovnkube-node-kptck\" (UID: \"6f805291-3eab-499d-aed8-577d64cf5c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:02.302528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.302509    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x55vs\" (UniqueName: \"kubernetes.io/projected/e8fa85d1-6827-43dd-a947-71d6ebc1ce6d-kube-api-access-x55vs\") pod \"ingress-canary-lgv9k\" (UID: \"e8fa85d1-6827-43dd-a947-71d6ebc1ce6d\") " pod="openshift-ingress-canary/ingress-canary-lgv9k"
Oct 30 23:19:02.303134 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.303113    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkvp\" (UniqueName: \"kubernetes.io/projected/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-api-access-jpkvp\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.303459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.303441    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t869h\" (UniqueName: \"kubernetes.io/projected/78e7226a-8a87-4a93-8c7b-642ff889d7d0-kube-api-access-t869h\") pod \"migrator-66df5d496b-58gzz\" (UID: \"78e7226a-8a87-4a93-8c7b-642ff889d7d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-58gzz"
Oct 30 23:19:02.303731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.303711    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4rz\" (UniqueName: \"kubernetes.io/projected/82ca7641-8711-4a48-86e0-a59f3ca8e498-kube-api-access-jf4rz\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:02.303799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.303784    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lstnh\" (UniqueName: \"kubernetes.io/projected/8919fd44-7b88-4215-9d80-50cf73eb1de7-kube-api-access-lstnh\") pod \"service-ca-5fffc97bdf-rhc9c\" (UID: \"8919fd44-7b88-4215-9d80-50cf73eb1de7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rhc9c"
Oct 30 23:19:02.303904 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.303890    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mffv\" (UniqueName: \"kubernetes.io/projected/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-kube-api-access-5mffv\") pod \"periodic-gathering-vlbq7-jsbgj\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") " pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:19:02.390750 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.390711    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:02.390933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.390774    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-crio-socket\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.390933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.390807    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.390933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.390834    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.390933 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.390869    4555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:19:02.391104 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.390868    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-crio-socket\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.391104 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.390955    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls podName:82ca7641-8711-4a48-86e0-a59f3ca8e498 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:02.890935522 +0000 UTC m=+1.817573096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls") pod "dns-default-b4l22" (UID: "82ca7641-8711-4a48-86e0-a59f3ca8e498") : secret "dns-default-metrics-tls" not found
Oct 30 23:19:02.391104 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.390960    4555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:19:02.391104 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.391021    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:02.891003448 +0000 UTC m=+1.817641008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:19:02.391447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.391430    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.793608 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.793566    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:19:02.793608 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.793589    4555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 23:19:02.793790 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.793659    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs podName:99889cd4-347f-46b6-938f-0886271bb04d nodeName:}" failed. No retries permitted until 2026-10-30 23:19:03.793646743 +0000 UTC m=+2.720284313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs") pod "network-metrics-daemon-vb7vz" (UID: "99889cd4-347f-46b6-938f-0886271bb04d") : secret "metrics-daemon-secret" not found
Oct 30 23:19:02.894568 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.894531    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:02.894750 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:02.894611    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:02.894750 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.894698    4555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:19:02.894750 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.894698    4555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:19:02.894861 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.894765    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls podName:82ca7641-8711-4a48-86e0-a59f3ca8e498 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:03.894748267 +0000 UTC m=+2.821385840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls") pod "dns-default-b4l22" (UID: "82ca7641-8711-4a48-86e0-a59f3ca8e498") : secret "dns-default-metrics-tls" not found
Oct 30 23:19:02.894861 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:02.894779    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:03.894773511 +0000 UTC m=+2.821411070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:19:03.802241 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:03.802193    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:19:03.802703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:03.802682    4555 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 23:19:03.804858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:03.804833    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99889cd4-347f-46b6-938f-0886271bb04d-metrics-certs\") pod \"network-metrics-daemon-vb7vz\" (UID: \"99889cd4-347f-46b6-938f-0886271bb04d\") " pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:19:03.903183 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:03.903147    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:03.903358 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:03.903207    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:03.903358 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:03.903297    4555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:19:03.903358 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:03.903316    4555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:19:03.903558 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:03.903351    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls podName:82ca7641-8711-4a48-86e0-a59f3ca8e498 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:05.903338094 +0000 UTC m=+4.829975666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls") pod "dns-default-b4l22" (UID: "82ca7641-8711-4a48-86e0-a59f3ca8e498") : secret "dns-default-metrics-tls" not found
Oct 30 23:19:03.903558 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:03.903519    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:05.903498294 +0000 UTC m=+4.830135856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:19:03.963491 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:03.963459    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb7vz"
Oct 30 23:19:04.268672 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:04.268582    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vb7vz"]
Oct 30 23:19:04.272410 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:04.272363    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99889cd4_347f_46b6_938f_0886271bb04d.slice/crio-1e52669c75bbea600bfe50105a92f962b77dd0d6aeec448ee28018a5550b4841 WatchSource:0}: Error finding container 1e52669c75bbea600bfe50105a92f962b77dd0d6aeec448ee28018a5550b4841: Status 404 returned error can't find the container with id 1e52669c75bbea600bfe50105a92f962b77dd0d6aeec448ee28018a5550b4841
Oct 30 23:19:04.275182 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:04.275166    4555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:19:05.255300 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:05.255244    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vb7vz" event={"ID":"99889cd4-347f-46b6-938f-0886271bb04d","Type":"ContainerStarted","Data":"1e52669c75bbea600bfe50105a92f962b77dd0d6aeec448ee28018a5550b4841"}
Oct 30 23:19:05.277436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:05.277413    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-czzsf_fea866f2-91be-418e-8a5f-dc609e36cfd5/dns-node-resolver/0.log"
Oct 30 23:19:05.917315 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:05.917276    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:05.917493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:05.917325    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:05.917493 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:05.917447    4555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:19:05.917493 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:05.917469    4555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:19:05.917603 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:05.917506    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls podName:82ca7641-8711-4a48-86e0-a59f3ca8e498 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:09.917490556 +0000 UTC m=+8.844128132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls") pod "dns-default-b4l22" (UID: "82ca7641-8711-4a48-86e0-a59f3ca8e498") : secret "dns-default-metrics-tls" not found
Oct 30 23:19:05.917603 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:05.917529    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:09.917514718 +0000 UTC m=+8.844152302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:19:06.077195 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:06.077167    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-prlzd_6748f5ad-4980-4e4b-b448-bdc7f50e9e57/node-ca/0.log"
Oct 30 23:19:06.260932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:06.260845    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vb7vz" event={"ID":"99889cd4-347f-46b6-938f-0886271bb04d","Type":"ContainerStarted","Data":"a0f3c9d537e66be87d5bec478b7e80a04ab2f37e2366e749db43e8a01fbc78b8"}
Oct 30 23:19:06.260932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:06.260901    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vb7vz" event={"ID":"99889cd4-347f-46b6-938f-0886271bb04d","Type":"ContainerStarted","Data":"539c9aeca765e4446ae36f9e84fd6b815d661bc92d00230bec5a317e351d8986"}
Oct 30 23:19:07.077141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:07.077067    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lgv9k_e8fa85d1-6827-43dd-a947-71d6ebc1ce6d/serve-healthcheck-canary/0.log"
Oct 30 23:19:07.277592 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:07.277560    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-58gzz_78e7226a-8a87-4a93-8c7b-642ff889d7d0/migrator/0.log"
Oct 30 23:19:08.897782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:08.897746    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:19:09.648028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:09.648000    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:09.944718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:09.944627    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:09.944718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:09.944681    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:09.945186 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:09.944782    4555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:19:09.945186 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:09.944838    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls podName:5a1dd390-1d71-4270-9a6f-d9e1c66c6e52 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:17.944824237 +0000 UTC m=+16.871461797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wk72x" (UID: "5a1dd390-1d71-4270-9a6f-d9e1c66c6e52") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:19:09.947198 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:09.947173    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82ca7641-8711-4a48-86e0-a59f3ca8e498-metrics-tls\") pod \"dns-default-b4l22\" (UID: \"82ca7641-8711-4a48-86e0-a59f3ca8e498\") " pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:09.967657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:09.967622    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:10.240003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:10.239851    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b4l22"]
Oct 30 23:19:10.243740 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:10.243717    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ca7641_8711_4a48_86e0_a59f3ca8e498.slice/crio-37d566f15813b6eaa9c010136fee824e5f1e4fc106d4b092884ce574bef61906 WatchSource:0}: Error finding container 37d566f15813b6eaa9c010136fee824e5f1e4fc106d4b092884ce574bef61906: Status 404 returned error can't find the container with id 37d566f15813b6eaa9c010136fee824e5f1e4fc106d4b092884ce574bef61906
Oct 30 23:19:10.274848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:10.274811    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4l22" event={"ID":"82ca7641-8711-4a48-86e0-a59f3ca8e498","Type":"ContainerStarted","Data":"37d566f15813b6eaa9c010136fee824e5f1e4fc106d4b092884ce574bef61906"}
Oct 30 23:19:10.741863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:10.741819    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:10.742067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:10.742055    4555 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:19:12.284586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:12.284495    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4l22" event={"ID":"82ca7641-8711-4a48-86e0-a59f3ca8e498","Type":"ContainerStarted","Data":"0875dbd012b9528e51c4dc9a0052f88c4b66c174990e736ad73d55519b91c318"}
Oct 30 23:19:12.284586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:12.284582    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4l22" event={"ID":"82ca7641-8711-4a48-86e0-a59f3ca8e498","Type":"ContainerStarted","Data":"2ed6cd076cd47188fcfb49b1138fbeaa4b92d9ded852245e5ec86900a2ddd613"}
Oct 30 23:19:12.285099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:12.284619    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:18.006045 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:18.006005    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:18.008416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:18.008398    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1dd390-1d71-4270-9a6f-d9e1c66c6e52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wk72x\" (UID: \"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52\") " pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:18.068153 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:18.068111    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wk72x"
Oct 30 23:19:18.339531 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:18.339499    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wk72x"]
Oct 30 23:19:18.343573 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:18.343544    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1dd390_1d71_4270_9a6f_d9e1c66c6e52.slice/crio-16bb0e2a1fba831bcb03ad7953c5b4cd544d628486f365412150b03298095648 WatchSource:0}: Error finding container 16bb0e2a1fba831bcb03ad7953c5b4cd544d628486f365412150b03298095648: Status 404 returned error can't find the container with id 16bb0e2a1fba831bcb03ad7953c5b4cd544d628486f365412150b03298095648
Oct 30 23:19:19.310307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:19.310283    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wk72x" event={"ID":"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52","Type":"ContainerStarted","Data":"dd897c692ec2d4640fb77bc8ede82ca0a09cb361d8473a88951c3d5e24f61cca"}
Oct 30 23:19:19.310616 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:19.310335    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wk72x" event={"ID":"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52","Type":"ContainerStarted","Data":"16bb0e2a1fba831bcb03ad7953c5b4cd544d628486f365412150b03298095648"}
Oct 30 23:19:20.196079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.196047    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lcxv4-ckdck"]
Oct 30 23:19:20.204393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.204346    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.218239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.218207    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lcxv4-ckdck"]
Oct 30 23:19:20.222139 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.222111    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-6hgnc"]
Oct 30 23:19:20.225127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.225107    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6hgnc"
Oct 30 23:19:20.227022 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.226997    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8589549b96-ztxc9"]
Oct 30 23:19:20.230051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.230029    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.235014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.234995    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 23:19:20.235127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.235114    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-nxn9v\""
Oct 30 23:19:20.235196 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.235186    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 23:19:20.244996 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.244974    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6hgnc"]
Oct 30 23:19:20.245587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.245560    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8589549b96-ztxc9"]
Oct 30 23:19:20.259210 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.259189    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 23:19:20.259332 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.259209    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 23:19:20.259647 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.259630    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 23:19:20.312256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.312220    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-757d6667b6-sshfm"]
Oct 30 23:19:20.315320 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.315294    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.316860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.316835    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wk72x" event={"ID":"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52","Type":"ContainerStarted","Data":"03e9d56d3b89691c1210f580c645863332513a7a0e195eff095af0442ed51203"}
Oct 30 23:19:20.321784 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.321767    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-oauth-serving-cert\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.321897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.321799    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-service-ca-bundle\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.321897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.321819    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnbkw\" (UniqueName: \"kubernetes.io/projected/2d37f12b-4177-44dd-8d90-a38eca561963-kube-api-access-fnbkw\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.321897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.321841    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec142d60-8f90-44f0-a57f-4de2294a028c-tmp\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.322144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.322030    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-trusted-ca-bundle\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.322144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.322071    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jw2g\" (UniqueName: \"kubernetes.io/projected/ec142d60-8f90-44f0-a57f-4de2294a028c-kube-api-access-2jw2g\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.322278 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.322159    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699kn\" (UniqueName: \"kubernetes.io/projected/630c5695-73be-4621-b26c-523270f26c14-kube-api-access-699kn\") pod \"downloads-6dd789c8b7-6hgnc\" (UID: \"630c5695-73be-4621-b26c-523270f26c14\") " pod="openshift-console/downloads-6dd789c8b7-6hgnc"
Oct 30 23:19:20.322278 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.322203    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630c5695-73be-4621-b26c-523270f26c14-tmp\") pod \"downloads-6dd789c8b7-6hgnc\" (UID: \"630c5695-73be-4621-b26c-523270f26c14\") " pod="openshift-console/downloads-6dd789c8b7-6hgnc"
Oct 30 23:19:20.322407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.322270    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-serving-cert\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.322407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.322317    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-console-config\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.322407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.322357    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-oauth-config\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.322564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.322430    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d37f12b-4177-44dd-8d90-a38eca561963-archives-path\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.322564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.322548    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-service-ca\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.325599 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.325579    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 23:19:20.335543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.335523    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757d6667b6-sshfm"]
Oct 30 23:19:20.341400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.340959    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gddrl\""
Oct 30 23:19:20.341400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.341037    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 23:19:20.341400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.341099    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 23:19:20.341400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.341137    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tr4dk\""
Oct 30 23:19:20.341400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.341185    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 23:19:20.341400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.341217    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 23:19:20.341400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.341278    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 23:19:20.423595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.423567    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-service-ca-bundle\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.423785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.423605    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbkw\" (UniqueName: \"kubernetes.io/projected/2d37f12b-4177-44dd-8d90-a38eca561963-kube-api-access-fnbkw\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.423785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.423633    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec142d60-8f90-44f0-a57f-4de2294a028c-tmp\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.423785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.423665    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/914db174-8917-48c4-b4d2-c6cab87225a4-ca-trust-extracted\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.423960 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.423821    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-trusted-ca-bundle\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.423960 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.423880    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jw2g\" (UniqueName: \"kubernetes.io/projected/ec142d60-8f90-44f0-a57f-4de2294a028c-kube-api-access-2jw2g\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.423960 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.423906    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/914db174-8917-48c4-b4d2-c6cab87225a4-installation-pull-secrets\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.423960 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.423936    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699kn\" (UniqueName: \"kubernetes.io/projected/630c5695-73be-4621-b26c-523270f26c14-kube-api-access-699kn\") pod \"downloads-6dd789c8b7-6hgnc\" (UID: \"630c5695-73be-4621-b26c-523270f26c14\") " pod="openshift-console/downloads-6dd789c8b7-6hgnc"
Oct 30 23:19:20.424192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.423967    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630c5695-73be-4621-b26c-523270f26c14-tmp\") pod \"downloads-6dd789c8b7-6hgnc\" (UID: \"630c5695-73be-4621-b26c-523270f26c14\") " pod="openshift-console/downloads-6dd789c8b7-6hgnc"
Oct 30 23:19:20.424192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.423996    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-serving-cert\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.424192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424030    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/914db174-8917-48c4-b4d2-c6cab87225a4-registry-tls\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.424192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424058    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/914db174-8917-48c4-b4d2-c6cab87225a4-image-registry-private-configuration\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.424192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424064    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec142d60-8f90-44f0-a57f-4de2294a028c-tmp\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.424192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424084    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sph8v\" (UniqueName: \"kubernetes.io/projected/914db174-8917-48c4-b4d2-c6cab87225a4-kube-api-access-sph8v\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.424192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424114    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-console-config\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.424192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424149    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-oauth-config\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.424192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424191    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d37f12b-4177-44dd-8d90-a38eca561963-archives-path\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.424788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424250    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/914db174-8917-48c4-b4d2-c6cab87225a4-trusted-ca\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.424788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424289    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-service-ca\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.424788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424320    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-oauth-serving-cert\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.424788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424352    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/914db174-8917-48c4-b4d2-c6cab87225a4-bound-sa-token\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.424788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424401    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/914db174-8917-48c4-b4d2-c6cab87225a4-registry-certificates\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.424788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424540    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d37f12b-4177-44dd-8d90-a38eca561963-archives-path\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.424788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424685    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-service-ca-bundle\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.424788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424760    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-trusted-ca-bundle\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.425207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424867    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630c5695-73be-4621-b26c-523270f26c14-tmp\") pod \"downloads-6dd789c8b7-6hgnc\" (UID: \"630c5695-73be-4621-b26c-523270f26c14\") " pod="openshift-console/downloads-6dd789c8b7-6hgnc"
Oct 30 23:19:20.425207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.424908    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-console-config\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.425317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.425278    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-service-ca\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.425553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.425510    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-oauth-serving-cert\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.427932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.427909    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-serving-cert\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.428939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.428908    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-oauth-config\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.439690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.439669    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbkw\" (UniqueName: \"kubernetes.io/projected/2d37f12b-4177-44dd-8d90-a38eca561963-kube-api-access-fnbkw\") pod \"periodic-gathering-lcxv4-ckdck\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") " pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.439811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.439782    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jw2g\" (UniqueName: \"kubernetes.io/projected/ec142d60-8f90-44f0-a57f-4de2294a028c-kube-api-access-2jw2g\") pod \"console-8589549b96-ztxc9\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") " pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.440316 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.440296    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-699kn\" (UniqueName: \"kubernetes.io/projected/630c5695-73be-4621-b26c-523270f26c14-kube-api-access-699kn\") pod \"downloads-6dd789c8b7-6hgnc\" (UID: \"630c5695-73be-4621-b26c-523270f26c14\") " pod="openshift-console/downloads-6dd789c8b7-6hgnc"
Oct 30 23:19:20.518162 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.518084    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:19:20.525659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.525631    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/914db174-8917-48c4-b4d2-c6cab87225a4-ca-trust-extracted\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.525789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.525681    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/914db174-8917-48c4-b4d2-c6cab87225a4-installation-pull-secrets\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.525789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.525719    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/914db174-8917-48c4-b4d2-c6cab87225a4-registry-tls\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.525789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.525749    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/914db174-8917-48c4-b4d2-c6cab87225a4-image-registry-private-configuration\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.525789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.525771    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sph8v\" (UniqueName: \"kubernetes.io/projected/914db174-8917-48c4-b4d2-c6cab87225a4-kube-api-access-sph8v\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.526002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.525807    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/914db174-8917-48c4-b4d2-c6cab87225a4-trusted-ca\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.526002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.525839    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/914db174-8917-48c4-b4d2-c6cab87225a4-bound-sa-token\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.526002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.525923    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/914db174-8917-48c4-b4d2-c6cab87225a4-registry-certificates\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.526574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.526512    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/914db174-8917-48c4-b4d2-c6cab87225a4-ca-trust-extracted\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.526846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.526821    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/914db174-8917-48c4-b4d2-c6cab87225a4-registry-certificates\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.527125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.526921    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/914db174-8917-48c4-b4d2-c6cab87225a4-trusted-ca\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.528782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.528760    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/914db174-8917-48c4-b4d2-c6cab87225a4-registry-tls\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.529112 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.529091    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/914db174-8917-48c4-b4d2-c6cab87225a4-image-registry-private-configuration\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.529635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.529608    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/914db174-8917-48c4-b4d2-c6cab87225a4-installation-pull-secrets\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.538717 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.538694    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/914db174-8917-48c4-b4d2-c6cab87225a4-bound-sa-token\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.539069 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.539050    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sph8v\" (UniqueName: \"kubernetes.io/projected/914db174-8917-48c4-b4d2-c6cab87225a4-kube-api-access-sph8v\") pod \"image-registry-757d6667b6-sshfm\" (UID: \"914db174-8917-48c4-b4d2-c6cab87225a4\") " pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.540854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.540833    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6hgnc"
Oct 30 23:19:20.548575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.548402    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:20.628986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.628954    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:20.802070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.802031    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lcxv4-ckdck"]
Oct 30 23:19:20.806434 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:20.806401    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d37f12b_4177_44dd_8d90_a38eca561963.slice/crio-65cea0b9e303b01548517c69a5ddfdd88aa6d5509593406a8bd8c4798ce7d0a8 WatchSource:0}: Error finding container 65cea0b9e303b01548517c69a5ddfdd88aa6d5509593406a8bd8c4798ce7d0a8: Status 404 returned error can't find the container with id 65cea0b9e303b01548517c69a5ddfdd88aa6d5509593406a8bd8c4798ce7d0a8
Oct 30 23:19:20.835323 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.835300    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6hgnc"]
Oct 30 23:19:20.839280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.839262    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8589549b96-ztxc9"]
Oct 30 23:19:20.840086 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:20.840060    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630c5695_73be_4621_b26c_523270f26c14.slice/crio-8f9b495b5579566d7dd84fbe8675e9ea4e8969f3cbf4447d898c104b3d8d8f43 WatchSource:0}: Error finding container 8f9b495b5579566d7dd84fbe8675e9ea4e8969f3cbf4447d898c104b3d8d8f43: Status 404 returned error can't find the container with id 8f9b495b5579566d7dd84fbe8675e9ea4e8969f3cbf4447d898c104b3d8d8f43
Oct 30 23:19:20.841938 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:20.841912    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec142d60_8f90_44f0_a57f_4de2294a028c.slice/crio-704049d00f8a431b9807f43ebaaf452b5f64a4ff33dcc581f87172dd4bd038dc WatchSource:0}: Error finding container 704049d00f8a431b9807f43ebaaf452b5f64a4ff33dcc581f87172dd4bd038dc: Status 404 returned error can't find the container with id 704049d00f8a431b9807f43ebaaf452b5f64a4ff33dcc581f87172dd4bd038dc
Oct 30 23:19:20.911969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:20.911934    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757d6667b6-sshfm"]
Oct 30 23:19:20.915800 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:20.915771    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914db174_8917_48c4_b4d2_c6cab87225a4.slice/crio-799961ead2d685028abdcce1a3b6808e28da989c662c1551b19009e8a2283c6f WatchSource:0}: Error finding container 799961ead2d685028abdcce1a3b6808e28da989c662c1551b19009e8a2283c6f: Status 404 returned error can't find the container with id 799961ead2d685028abdcce1a3b6808e28da989c662c1551b19009e8a2283c6f
Oct 30 23:19:21.025949 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.025918    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp"]
Oct 30 23:19:21.029451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.029424    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp"
Oct 30 23:19:21.039622 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.039593    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-49flk\""
Oct 30 23:19:21.039838 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.039822    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 23:19:21.039962 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.039936    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp"]
Oct 30 23:19:21.130297 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.130218    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5dfa02be-1cdd-462a-bdec-b5c0c182e1f4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-74csp\" (UID: \"5dfa02be-1cdd-462a-bdec-b5c0c182e1f4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp"
Oct 30 23:19:21.231424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.231383    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5dfa02be-1cdd-462a-bdec-b5c0c182e1f4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-74csp\" (UID: \"5dfa02be-1cdd-462a-bdec-b5c0c182e1f4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp"
Oct 30 23:19:21.233900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.233879    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5dfa02be-1cdd-462a-bdec-b5c0c182e1f4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-74csp\" (UID: \"5dfa02be-1cdd-462a-bdec-b5c0c182e1f4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp"
Oct 30 23:19:21.321840 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.321802    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757d6667b6-sshfm" event={"ID":"914db174-8917-48c4-b4d2-c6cab87225a4","Type":"ContainerStarted","Data":"f528db5138dfb1f3c483f9eaadf46602c2f0011e7295e7df5b90176277804551"}
Oct 30 23:19:21.322276 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.321858    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757d6667b6-sshfm" event={"ID":"914db174-8917-48c4-b4d2-c6cab87225a4","Type":"ContainerStarted","Data":"799961ead2d685028abdcce1a3b6808e28da989c662c1551b19009e8a2283c6f"}
Oct 30 23:19:21.322276 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.321995    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:21.323084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.323061    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8589549b96-ztxc9" event={"ID":"ec142d60-8f90-44f0-a57f-4de2294a028c","Type":"ContainerStarted","Data":"704049d00f8a431b9807f43ebaaf452b5f64a4ff33dcc581f87172dd4bd038dc"}
Oct 30 23:19:21.324163 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.324140    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6hgnc" event={"ID":"630c5695-73be-4621-b26c-523270f26c14","Type":"ContainerStarted","Data":"8f9b495b5579566d7dd84fbe8675e9ea4e8969f3cbf4447d898c104b3d8d8f43"}
Oct 30 23:19:21.326318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.326296    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wk72x" event={"ID":"5a1dd390-1d71-4270-9a6f-d9e1c66c6e52","Type":"ContainerStarted","Data":"63063c932fc85df4946ab7a9999d2d50663c844dc4e4a4fa78bf831a972ec0b4"}
Oct 30 23:19:21.327770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.327742    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lcxv4-ckdck" event={"ID":"2d37f12b-4177-44dd-8d90-a38eca561963","Type":"ContainerStarted","Data":"c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c"}
Oct 30 23:19:21.327840 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.327779    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lcxv4-ckdck" event={"ID":"2d37f12b-4177-44dd-8d90-a38eca561963","Type":"ContainerStarted","Data":"65cea0b9e303b01548517c69a5ddfdd88aa6d5509593406a8bd8c4798ce7d0a8"}
Oct 30 23:19:21.346093 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.346043    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-757d6667b6-sshfm" podStartSLOduration=1.346031404 podStartE2EDuration="1.346031404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:21.345176664 +0000 UTC m=+20.271814246" watchObservedRunningTime="2026-10-30 23:19:21.346031404 +0000 UTC m=+20.272668984"
Oct 30 23:19:21.353963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.353942    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp"
Oct 30 23:19:21.387621 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.387519    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lcxv4-ckdck" podStartSLOduration=1.387504687 podStartE2EDuration="1.387504687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:21.387011875 +0000 UTC m=+20.313649456" watchObservedRunningTime="2026-10-30 23:19:21.387504687 +0000 UTC m=+20.314142262"
Oct 30 23:19:21.629492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:21.629459    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp"]
Oct 30 23:19:21.634144 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:21.634113    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfa02be_1cdd_462a_bdec_b5c0c182e1f4.slice/crio-33a8e6aedf4c6b2bfbbe6bc85c9e12729b18d04231e3aeab6478fc8799a1748d WatchSource:0}: Error finding container 33a8e6aedf4c6b2bfbbe6bc85c9e12729b18d04231e3aeab6478fc8799a1748d: Status 404 returned error can't find the container with id 33a8e6aedf4c6b2bfbbe6bc85c9e12729b18d04231e3aeab6478fc8799a1748d
Oct 30 23:19:22.038627 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:22.038592    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4l22_82ca7641-8711-4a48-86e0-a59f3ca8e498/dns/0.log"
Oct 30 23:19:22.238006 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:22.237976    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4l22_82ca7641-8711-4a48-86e0-a59f3ca8e498/kube-rbac-proxy/0.log"
Oct 30 23:19:22.290167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:22.290083    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b4l22"
Oct 30 23:19:22.339429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:22.339383    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp" event={"ID":"5dfa02be-1cdd-462a-bdec-b5c0c182e1f4","Type":"ContainerStarted","Data":"33a8e6aedf4c6b2bfbbe6bc85c9e12729b18d04231e3aeab6478fc8799a1748d"}
Oct 30 23:19:22.638450 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:22.638263    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-czzsf_fea866f2-91be-418e-8a5f-dc609e36cfd5/dns-node-resolver/0.log"
Oct 30 23:19:23.238467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:23.238427    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-757d6667b6-sshfm_914db174-8917-48c4-b4d2-c6cab87225a4/registry/0.log"
Oct 30 23:19:23.346109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:23.345639    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp" event={"ID":"5dfa02be-1cdd-462a-bdec-b5c0c182e1f4","Type":"ContainerStarted","Data":"885bc331d8d8d58b0846fd9aabc6d49b6a144d5301436c80dcddd553da5dfba7"}
Oct 30 23:19:23.346109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:23.345820    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp"
Oct 30 23:19:23.352242 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:23.352211    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp"
Oct 30 23:19:23.364342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:23.364286    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-74csp" podStartSLOduration=0.780065509 podStartE2EDuration="2.364270974s" totalImagesPullingTime="1.584205465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:23.363794213 +0000 UTC m=+22.290431798" watchObservedRunningTime="2026-10-30 23:19:23.364270974 +0000 UTC m=+22.290908556"
Oct 30 23:19:23.637418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:23.637392    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-prlzd_6748f5ad-4980-4e4b-b448-bdc7f50e9e57/node-ca/0.log"
Oct 30 23:19:24.095522 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.094399    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"]
Oct 30 23:19:24.103899 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.103872    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.107563 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.107442    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"]
Oct 30 23:19:24.108393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.108326    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 23:19:24.109079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.109057    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 23:19:24.109079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.109068    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 23:19:24.109349 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.109333    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 23:19:24.109460 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.109428    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-j5j9s\""
Oct 30 23:19:24.109517 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.109459    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 23:19:24.257491 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.257455    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14801d31-6150-4b26-8429-b5a66d372117-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.257663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.257519    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14801d31-6150-4b26-8429-b5a66d372117-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.257737 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.257680    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d42jq\" (UniqueName: \"kubernetes.io/projected/14801d31-6150-4b26-8429-b5a66d372117-kube-api-access-d42jq\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.257799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.257734    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14801d31-6150-4b26-8429-b5a66d372117-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.352762 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.352672    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8589549b96-ztxc9" event={"ID":"ec142d60-8f90-44f0-a57f-4de2294a028c","Type":"ContainerStarted","Data":"163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad"}
Oct 30 23:19:24.358623 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.358599    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14801d31-6150-4b26-8429-b5a66d372117-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.358770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.358634    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d42jq\" (UniqueName: \"kubernetes.io/projected/14801d31-6150-4b26-8429-b5a66d372117-kube-api-access-d42jq\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.358770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.358673    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14801d31-6150-4b26-8429-b5a66d372117-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.358770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.358738    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14801d31-6150-4b26-8429-b5a66d372117-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.359443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.359421    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14801d31-6150-4b26-8429-b5a66d372117-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.361675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.361648    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14801d31-6150-4b26-8429-b5a66d372117-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.361765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.361658    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14801d31-6150-4b26-8429-b5a66d372117-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.371525 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.371499    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d42jq\" (UniqueName: \"kubernetes.io/projected/14801d31-6150-4b26-8429-b5a66d372117-kube-api-access-d42jq\") pod \"prometheus-operator-9cd89b74c-8k4pb\" (UID: \"14801d31-6150-4b26-8429-b5a66d372117\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.373668 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.373615    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8589549b96-ztxc9" podStartSLOduration=1.071076322 podStartE2EDuration="4.373597479s" totalImagesPullingTime="3.302521157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:24.372583536 +0000 UTC m=+23.299221112" watchObservedRunningTime="2026-10-30 23:19:24.373597479 +0000 UTC m=+23.300235062"
Oct 30 23:19:24.423333 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.423296    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"
Oct 30 23:19:24.638659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.638582    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lgv9k_e8fa85d1-6827-43dd-a947-71d6ebc1ce6d/serve-healthcheck-canary/0.log"
Oct 30 23:19:24.705087 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:24.705054    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb"]
Oct 30 23:19:24.709347 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:24.709318    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14801d31_6150_4b26_8429_b5a66d372117.slice/crio-64e593dfd5fb4f01192441c22a8cd3786b236824eddc5ecafee48561d0ebeab4 WatchSource:0}: Error finding container 64e593dfd5fb4f01192441c22a8cd3786b236824eddc5ecafee48561d0ebeab4: Status 404 returned error can't find the container with id 64e593dfd5fb4f01192441c22a8cd3786b236824eddc5ecafee48561d0ebeab4
Oct 30 23:19:25.046814 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:25.046783    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kptck"
Oct 30 23:19:25.357604 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:25.357520    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb" event={"ID":"14801d31-6150-4b26-8429-b5a66d372117","Type":"ContainerStarted","Data":"64e593dfd5fb4f01192441c22a8cd3786b236824eddc5ecafee48561d0ebeab4"}
Oct 30 23:19:26.364984 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:26.364950    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb" event={"ID":"14801d31-6150-4b26-8429-b5a66d372117","Type":"ContainerStarted","Data":"f9d3abff5f4db67346a7049b118d63447ef032b615e60383b89ea316a40ccb5f"}
Oct 30 23:19:26.365359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:26.365035    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb" event={"ID":"14801d31-6150-4b26-8429-b5a66d372117","Type":"ContainerStarted","Data":"7cb70f20dc113df3515b7b0e5314595c92e02b4522dacbadfaac418878605d0e"}
Oct 30 23:19:26.392416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:26.392344    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8k4pb" podStartSLOduration=0.946114767 podStartE2EDuration="2.392326947s" totalImagesPullingTime="1.44621218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:26.391743417 +0000 UTC m=+25.318381004" watchObservedRunningTime="2026-10-30 23:19:26.392326947 +0000 UTC m=+25.318964530"
Oct 30 23:19:27.451838 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:27.451796    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:19:28.021887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.021858    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l94rx"
Oct 30 23:19:28.489862 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.489784    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zjgtn"]
Oct 30 23:19:28.493279 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.493250    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.588860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.588817    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 23:19:28.589129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.589106    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hgj94\""
Oct 30 23:19:28.589194 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.589154    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 23:19:28.600077 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.600043    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-wtmp\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.600226 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.600090    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-textfile\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.600226 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.600132    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b05378f-b90e-47f8-9958-20da20c16505-metrics-client-ca\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.600226 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.600161    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-accelerators-collector-config\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.600226 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.600202    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b05378f-b90e-47f8-9958-20da20c16505-root\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.600488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.600234    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqkv\" (UniqueName: \"kubernetes.io/projected/3b05378f-b90e-47f8-9958-20da20c16505-kube-api-access-thqkv\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.600488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.600265    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b05378f-b90e-47f8-9958-20da20c16505-sys\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.600488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.600290    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.600488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.600322    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-tls\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.601600 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.601579    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 23:19:28.701661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.701625    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b05378f-b90e-47f8-9958-20da20c16505-metrics-client-ca\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.701955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.701681    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-accelerators-collector-config\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.701955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.701721    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b05378f-b90e-47f8-9958-20da20c16505-root\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.701955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.701750    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqkv\" (UniqueName: \"kubernetes.io/projected/3b05378f-b90e-47f8-9958-20da20c16505-kube-api-access-thqkv\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.701955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.701782    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b05378f-b90e-47f8-9958-20da20c16505-sys\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.701955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.701806    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.701955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.701842    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-tls\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.701955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.701870    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-wtmp\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.701955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.701901    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-textfile\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.702521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.702163    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b05378f-b90e-47f8-9958-20da20c16505-root\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.702521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.702170    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-textfile\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.702521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.702261    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b05378f-b90e-47f8-9958-20da20c16505-metrics-client-ca\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.702521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.702416    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-wtmp\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.702521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.702463    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b05378f-b90e-47f8-9958-20da20c16505-sys\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.703011 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.702989    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-accelerators-collector-config\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.705343 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.705323    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-tls\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.705835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.705809    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b05378f-b90e-47f8-9958-20da20c16505-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.721514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.721482    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqkv\" (UniqueName: \"kubernetes.io/projected/3b05378f-b90e-47f8-9958-20da20c16505-kube-api-access-thqkv\") pod \"node-exporter-zjgtn\" (UID: \"3b05378f-b90e-47f8-9958-20da20c16505\") " pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.813489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.813200    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zjgtn"
Oct 30 23:19:28.832797 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:28.828543    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b05378f_b90e_47f8_9958_20da20c16505.slice/crio-fdc039bcc391b724b209dd83677c027f65ad9626f738c9d218e20ab60480cd15 WatchSource:0}: Error finding container fdc039bcc391b724b209dd83677c027f65ad9626f738c9d218e20ab60480cd15: Status 404 returned error can't find the container with id fdc039bcc391b724b209dd83677c027f65ad9626f738c9d218e20ab60480cd15
Oct 30 23:19:28.901001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:28.900970    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vzvtj"
Oct 30 23:19:29.377688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:29.377648    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zjgtn" event={"ID":"3b05378f-b90e-47f8-9958-20da20c16505","Type":"ContainerStarted","Data":"fdc039bcc391b724b209dd83677c027f65ad9626f738c9d218e20ab60480cd15"}
Oct 30 23:19:30.383719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:30.383682    4555 generic.go:356] "Generic (PLEG): container finished" podID="3b05378f-b90e-47f8-9958-20da20c16505" containerID="f2e84d43b47cebb2677c9b3360231d41bc643c58dfeafc544c3555bfd6cab10f" exitCode=0
Oct 30 23:19:30.384142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:30.383741    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zjgtn" event={"ID":"3b05378f-b90e-47f8-9958-20da20c16505","Type":"ContainerDied","Data":"f2e84d43b47cebb2677c9b3360231d41bc643c58dfeafc544c3555bfd6cab10f"}
Oct 30 23:19:30.549210 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:30.549174    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:30.549398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:30.549277    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:30.556272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:30.556244    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:31.405886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:31.405855    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zjgtn" event={"ID":"3b05378f-b90e-47f8-9958-20da20c16505","Type":"ContainerStarted","Data":"a2c93c54c906c208819ab0551dbebe48ac63930474594a5116eb52860d777356"}
Oct 30 23:19:31.406273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:31.405905    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zjgtn" event={"ID":"3b05378f-b90e-47f8-9958-20da20c16505","Type":"ContainerStarted","Data":"36d981e7d5bdefe21d53800812534e8d2ccdd0b39eb7361f4e0ca461ef18ec95"}
Oct 30 23:19:31.410622 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:31.410598    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:19:31.430702 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:31.430637    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zjgtn" podStartSLOduration=2.023671451 podStartE2EDuration="3.430618922s" totalImagesPullingTime="1.318023419s" totalInitContainerRuntime="88.924052ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:31.429045538 +0000 UTC m=+30.355683119" watchObservedRunningTime="2026-10-30 23:19:31.430618922 +0000 UTC m=+30.357256504"
Oct 30 23:19:35.609580 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.609515    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b898fd4b9-9mvfg"]
Oct 30 23:19:35.644228 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.644193    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b898fd4b9-9mvfg"]
Oct 30 23:19:35.644416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.644330    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.652694 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.652662    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 23:19:35.761805 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.761765    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-oauth-serving-cert\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.761980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.761863    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a79f63-2cb9-4102-9abf-6d431267b1bf-tmp\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.761980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.761927    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-trusted-ca-bundle\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.761980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.761965    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-service-ca\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.762099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.762062    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-oauth-config\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.762155 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.762112    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-serving-cert\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.762200 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.762180    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5864h\" (UniqueName: \"kubernetes.io/projected/62a79f63-2cb9-4102-9abf-6d431267b1bf-kube-api-access-5864h\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.762237 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.762206    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-config\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.863273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.863192    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-serving-cert\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.863273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.863236    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5864h\" (UniqueName: \"kubernetes.io/projected/62a79f63-2cb9-4102-9abf-6d431267b1bf-kube-api-access-5864h\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.863273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.863268    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-config\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.863565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.863302    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-oauth-serving-cert\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.863565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.863332    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a79f63-2cb9-4102-9abf-6d431267b1bf-tmp\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.863565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.863363    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-trusted-ca-bundle\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.863565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.863405    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-service-ca\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.863565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.863447    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-oauth-config\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.864433 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.864400    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-service-ca\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.864572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.864543    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-config\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.864739 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.864718    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-trusted-ca-bundle\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.864813 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.864723    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-oauth-serving-cert\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.864889 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.864864    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a79f63-2cb9-4102-9abf-6d431267b1bf-tmp\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.866344 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.866313    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-oauth-config\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.866636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.866613    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-serving-cert\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.876346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.876320    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5864h\" (UniqueName: \"kubernetes.io/projected/62a79f63-2cb9-4102-9abf-6d431267b1bf-kube-api-access-5864h\") pod \"console-7b898fd4b9-9mvfg\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") " pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:35.972843 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:35.972802    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:36.288010 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:36.287978    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b898fd4b9-9mvfg"]
Oct 30 23:19:36.292695 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:36.292654    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a79f63_2cb9_4102_9abf_6d431267b1bf.slice/crio-8682937ce9e37f3dcf1cee1cd94adf59ac5c8f0729674b374ca13f16663f2a60 WatchSource:0}: Error finding container 8682937ce9e37f3dcf1cee1cd94adf59ac5c8f0729674b374ca13f16663f2a60: Status 404 returned error can't find the container with id 8682937ce9e37f3dcf1cee1cd94adf59ac5c8f0729674b374ca13f16663f2a60
Oct 30 23:19:36.428137 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:36.428096    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b898fd4b9-9mvfg" event={"ID":"62a79f63-2cb9-4102-9abf-6d431267b1bf","Type":"ContainerStarted","Data":"6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9"}
Oct 30 23:19:36.428329 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:36.428162    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b898fd4b9-9mvfg" event={"ID":"62a79f63-2cb9-4102-9abf-6d431267b1bf","Type":"ContainerStarted","Data":"8682937ce9e37f3dcf1cee1cd94adf59ac5c8f0729674b374ca13f16663f2a60"}
Oct 30 23:19:36.452601 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:36.451995    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b898fd4b9-9mvfg" podStartSLOduration=1.451975948 podStartE2EDuration="1.451975948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:36.451068346 +0000 UTC m=+35.377705927" watchObservedRunningTime="2026-10-30 23:19:36.451975948 +0000 UTC m=+35.378613529"
Oct 30 23:19:39.442129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:39.442086    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6hgnc" event={"ID":"630c5695-73be-4621-b26c-523270f26c14","Type":"ContainerStarted","Data":"b7781718ab356efc73cd75bed91b4a2b72bfb2c2f727fe585d6f885ff91454c5"}
Oct 30 23:19:39.442762 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:39.442289    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-6hgnc"
Oct 30 23:19:39.461937 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:39.461908    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-6hgnc"
Oct 30 23:19:39.491728 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:39.491676    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-6hgnc" podStartSLOduration=1.9639071559999999 podStartE2EDuration="19.491657116s" totalImagesPullingTime="17.52774996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:39.490865032 +0000 UTC m=+38.417502631" watchObservedRunningTime="2026-10-30 23:19:39.491657116 +0000 UTC m=+38.418294701"
Oct 30 23:19:40.718451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.718413    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b898fd4b9-9mvfg"]
Oct 30 23:19:40.758351 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.758319    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b97c77649-8g86l"]
Oct 30 23:19:40.782436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.782407    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b97c77649-8g86l"]
Oct 30 23:19:40.782611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.782526    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.814362 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.814326    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-serving-cert\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.814562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.814428    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-config\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.814562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.814465    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-service-ca\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.814562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.814537    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-oauth-serving-cert\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.814712 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.814580    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-trusted-ca-bundle\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.814712 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.814611    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6342fbca-d47c-4d4f-8e65-0fcade1a2107-tmp\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.814712 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.814628    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpblg\" (UniqueName: \"kubernetes.io/projected/6342fbca-d47c-4d4f-8e65-0fcade1a2107-kube-api-access-mpblg\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.814820 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.814728    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-oauth-config\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.915155 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.915106    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-serving-cert\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.915339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.915165    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-config\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.915339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.915205    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-service-ca\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.915339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.915230    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-oauth-serving-cert\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.915339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.915259    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-trusted-ca-bundle\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.915339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.915294    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6342fbca-d47c-4d4f-8e65-0fcade1a2107-tmp\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.915339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.915322    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpblg\" (UniqueName: \"kubernetes.io/projected/6342fbca-d47c-4d4f-8e65-0fcade1a2107-kube-api-access-mpblg\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.915631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.915394    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-oauth-config\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.917534 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.916118    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6342fbca-d47c-4d4f-8e65-0fcade1a2107-tmp\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.917534 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.916441    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-oauth-serving-cert\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.917534 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.917193    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-service-ca\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.917534 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.917455    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-config\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.917534 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.917490    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-trusted-ca-bundle\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.918852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.918829    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-serving-cert\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.919961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.919938    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-oauth-config\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:40.928469 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:40.928438    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpblg\" (UniqueName: \"kubernetes.io/projected/6342fbca-d47c-4d4f-8e65-0fcade1a2107-kube-api-access-mpblg\") pod \"console-6b97c77649-8g86l\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") " pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:41.115113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:41.115071    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:41.454499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:41.447249    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b97c77649-8g86l"]
Oct 30 23:19:41.464601 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:19:41.464564    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6342fbca_d47c_4d4f_8e65_0fcade1a2107.slice/crio-fa7ba13bb4659b3b702fbbaf1c2385e530059ddd1663844dba17342b9542ff4b WatchSource:0}: Error finding container fa7ba13bb4659b3b702fbbaf1c2385e530059ddd1663844dba17342b9542ff4b: Status 404 returned error can't find the container with id fa7ba13bb4659b3b702fbbaf1c2385e530059ddd1663844dba17342b9542ff4b
Oct 30 23:19:42.344755 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:42.344724    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-757d6667b6-sshfm"
Oct 30 23:19:42.458520 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:42.458483    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b97c77649-8g86l" event={"ID":"6342fbca-d47c-4d4f-8e65-0fcade1a2107","Type":"ContainerStarted","Data":"c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5"}
Oct 30 23:19:42.458688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:42.458539    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b97c77649-8g86l" event={"ID":"6342fbca-d47c-4d4f-8e65-0fcade1a2107","Type":"ContainerStarted","Data":"fa7ba13bb4659b3b702fbbaf1c2385e530059ddd1663844dba17342b9542ff4b"}
Oct 30 23:19:42.480575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:42.480511    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b97c77649-8g86l" podStartSLOduration=2.480493138 podStartE2EDuration="2.480493138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:42.480109667 +0000 UTC m=+41.406747276" watchObservedRunningTime="2026-10-30 23:19:42.480493138 +0000 UTC m=+41.407130720"
Oct 30 23:19:45.972997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:45.972955    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:19:51.115691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:51.115629    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:51.116334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:51.115704    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:51.120467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:51.120442    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:51.497801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:51.497730    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:19:51.555803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:51.555768    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8589549b96-ztxc9"]
Oct 30 23:19:51.834227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:51.834182    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vlbq7-jsbgj"]
Oct 30 23:19:51.835117 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:51.834970    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-vlbq7-jsbgj" podUID="481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" containerName="insights-gathering" containerID="cri-o://d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e" gracePeriod=30
Oct 30 23:19:52.174732 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.174694    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:19:52.220877 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.220840    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-service-ca-bundle\") pod \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") "
Oct 30 23:19:52.221058 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.220913    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-archives-path\") pod \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") "
Oct 30 23:19:52.221058 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.220968    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-trusted-ca-bundle\") pod \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") "
Oct 30 23:19:52.221058 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.220999    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-kube-api-access-5mffv\" (UniqueName: \"kubernetes.io/projected/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-kube-api-access-5mffv\") pod \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\" (UID: \"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06\") "
Oct 30 23:19:52.221291 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.221253    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-service-ca-bundle" pod "481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" (UID: "481ae6f7-3ce8-4964-a9e2-60d26b2c6b06"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:19:52.221291 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.221217    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-archives-path" pod "481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" (UID: "481ae6f7-3ce8-4964-a9e2-60d26b2c6b06"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:52.221527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.221348    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-trusted-ca-bundle" pod "481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" (UID: "481ae6f7-3ce8-4964-a9e2-60d26b2c6b06"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:19:52.223356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.223334    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-kube-api-access-5mffv" pod "481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" (UID: "481ae6f7-3ce8-4964-a9e2-60d26b2c6b06"). InnerVolumeSpecName "kube-api-access-5mffv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:52.322407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.322348    4555 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-service-ca-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:52.322407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.322400    4555 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-archives-path\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:52.322407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.322410    4555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-trusted-ca-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:52.322663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.322419    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mffv\" (UniqueName: \"kubernetes.io/projected/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06-kube-api-access-5mffv\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:52.497044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.496950    4555 generic.go:356] "Generic (PLEG): container finished" podID="481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" containerID="d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e" exitCode=2
Oct 30 23:19:52.497044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.497008    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vlbq7-jsbgj"
Oct 30 23:19:52.497304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.497054    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vlbq7-jsbgj" event={"ID":"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06","Type":"ContainerDied","Data":"d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e"}
Oct 30 23:19:52.497304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.497103    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vlbq7-jsbgj" event={"ID":"481ae6f7-3ce8-4964-a9e2-60d26b2c6b06","Type":"ContainerDied","Data":"8eca9237ef2e72f8839e1d94ae7f7bed67f833431a8b64cc60fb4ce53d8acfe0"}
Oct 30 23:19:52.497304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.497143    4555 scope.go:122] "RemoveContainer" containerID="d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e"
Oct 30 23:19:52.505676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.505655    4555 scope.go:122] "RemoveContainer" containerID="d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e"
Oct 30 23:19:52.505978 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:52.505956    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e\": container with ID starting with d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e not found: ID does not exist" containerID="d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e"
Oct 30 23:19:52.506068 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.506005    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e"} err="failed to get container status \"d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e\": rpc error: code = NotFound desc = could not find container \"d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e\": container with ID starting with d49d39fa1f0e157abd715297937aae79baa497b804e02a2dc408c937a75e944e not found: ID does not exist"
Oct 30 23:19:52.518183 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.518160    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vlbq7-jsbgj"]
Oct 30 23:19:52.522565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:52.522544    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-vlbq7-jsbgj"]
Oct 30 23:19:53.204862 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:19:53.204825    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" path="/var/lib/kubelet/pods/481ae6f7-3ce8-4964-a9e2-60d26b2c6b06/volumes"
Oct 30 23:19:57.444676 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:19:57.444633    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:20:02.775630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.775594    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b99cfb955-4zgzj"]
Oct 30 23:20:02.776074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.775896    4555 cpu_manager.go:414] "containerMap: removing container" podUID="481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" containerName="insights-gathering"
Oct 30 23:20:02.776074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.775907    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" containerName="insights-gathering"
Oct 30 23:20:02.776074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.775965    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="481ae6f7-3ce8-4964-a9e2-60d26b2c6b06" containerName="insights-gathering"
Oct 30 23:20:02.778894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.778875    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:02.789209 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.789179    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b99cfb955-4zgzj"]
Oct 30 23:20:02.904847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.904807    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-oauth-serving-cert\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:02.904847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.904850    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9d87\" (UniqueName: \"kubernetes.io/projected/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-kube-api-access-q9d87\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:02.905082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.904873    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-service-ca\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:02.905082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.904899    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-oauth-config\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:02.905082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.904991    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-tmp\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:02.905184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.905088    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-config\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:02.905184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.905125    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-serving-cert\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:02.905248 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:02.905190    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-trusted-ca-bundle\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.006137 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.006091    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-oauth-serving-cert\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.006243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.006144    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9d87\" (UniqueName: \"kubernetes.io/projected/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-kube-api-access-q9d87\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.006243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.006176    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-service-ca\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.006243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.006212    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-oauth-config\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.006417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.006243    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-tmp\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.006417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.006278    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-config\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.006417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.006308    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-serving-cert\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.006417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.006352    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-trusted-ca-bundle\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.006897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.006862    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-tmp\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.007089 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.007036    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-service-ca\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.007206 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.007093    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-oauth-serving-cert\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.007206 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.007113    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-config\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.007206 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.007149    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-trusted-ca-bundle\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.008945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.008924    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-oauth-config\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.009045 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.009030    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-serving-cert\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.019244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.019204    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9d87\" (UniqueName: \"kubernetes.io/projected/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-kube-api-access-q9d87\") pod \"console-7b99cfb955-4zgzj\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") " pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.089265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.089175    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:03.362941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.362864    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b99cfb955-4zgzj"]
Oct 30 23:20:03.366471 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:20:03.366446    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8aa6bf5_2dc1_4b1f_97f3_78301e16fa5b.slice/crio-b8b98bd99caa8a5f71edb157b4506410a1ecc6de72b6317a60c5c199e9bd9bd4 WatchSource:0}: Error finding container b8b98bd99caa8a5f71edb157b4506410a1ecc6de72b6317a60c5c199e9bd9bd4: Status 404 returned error can't find the container with id b8b98bd99caa8a5f71edb157b4506410a1ecc6de72b6317a60c5c199e9bd9bd4
Oct 30 23:20:03.539472 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.539437    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b99cfb955-4zgzj" event={"ID":"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b","Type":"ContainerStarted","Data":"5d6c8ae79249c9d22eb75477e9b226393d0033864339d89de744aa16b24a4b7f"}
Oct 30 23:20:03.539687 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.539486    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b99cfb955-4zgzj" event={"ID":"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b","Type":"ContainerStarted","Data":"b8b98bd99caa8a5f71edb157b4506410a1ecc6de72b6317a60c5c199e9bd9bd4"}
Oct 30 23:20:03.562637 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:03.562586    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b99cfb955-4zgzj" podStartSLOduration=1.56257189 podStartE2EDuration="1.56257189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:03.562151586 +0000 UTC m=+62.488789168" watchObservedRunningTime="2026-10-30 23:20:03.56257189 +0000 UTC m=+62.489209472"
Oct 30 23:20:05.741423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:05.741347    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7b898fd4b9-9mvfg" podUID="62a79f63-2cb9-4102-9abf-6d431267b1bf" containerName="console" containerID="cri-o://6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9" gracePeriod=15
Oct 30 23:20:06.034151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.034122    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:20:06.133317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133286    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a79f63-2cb9-4102-9abf-6d431267b1bf-tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a79f63-2cb9-4102-9abf-6d431267b1bf-tmp\") pod \"62a79f63-2cb9-4102-9abf-6d431267b1bf\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") "
Oct 30 23:20:06.133506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133340    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-oauth-config\") pod \"62a79f63-2cb9-4102-9abf-6d431267b1bf\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") "
Oct 30 23:20:06.133506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133386    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-trusted-ca-bundle\") pod \"62a79f63-2cb9-4102-9abf-6d431267b1bf\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") "
Oct 30 23:20:06.133506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133408    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-service-ca\") pod \"62a79f63-2cb9-4102-9abf-6d431267b1bf\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") "
Oct 30 23:20:06.133506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133448    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-serving-cert\") pod \"62a79f63-2cb9-4102-9abf-6d431267b1bf\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") "
Oct 30 23:20:06.133506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133471    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-config\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-config\") pod \"62a79f63-2cb9-4102-9abf-6d431267b1bf\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") "
Oct 30 23:20:06.133506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133504    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-oauth-serving-cert\") pod \"62a79f63-2cb9-4102-9abf-6d431267b1bf\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") "
Oct 30 23:20:06.133797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133583    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a79f63-2cb9-4102-9abf-6d431267b1bf-kube-api-access-5864h\" (UniqueName: \"kubernetes.io/projected/62a79f63-2cb9-4102-9abf-6d431267b1bf-kube-api-access-5864h\") pod \"62a79f63-2cb9-4102-9abf-6d431267b1bf\" (UID: \"62a79f63-2cb9-4102-9abf-6d431267b1bf\") "
Oct 30 23:20:06.133797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133623    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a79f63-2cb9-4102-9abf-6d431267b1bf-tmp" pod "62a79f63-2cb9-4102-9abf-6d431267b1bf" (UID: "62a79f63-2cb9-4102-9abf-6d431267b1bf"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:20:06.133863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133810    4555 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a79f63-2cb9-4102-9abf-6d431267b1bf-tmp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:06.133916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.133895    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-trusted-ca-bundle" pod "62a79f63-2cb9-4102-9abf-6d431267b1bf" (UID: "62a79f63-2cb9-4102-9abf-6d431267b1bf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:06.134075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.134037    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-service-ca" pod "62a79f63-2cb9-4102-9abf-6d431267b1bf" (UID: "62a79f63-2cb9-4102-9abf-6d431267b1bf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:06.134181 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.134042    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-config" pod "62a79f63-2cb9-4102-9abf-6d431267b1bf" (UID: "62a79f63-2cb9-4102-9abf-6d431267b1bf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:06.134243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.134059    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-oauth-serving-cert" pod "62a79f63-2cb9-4102-9abf-6d431267b1bf" (UID: "62a79f63-2cb9-4102-9abf-6d431267b1bf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:06.135894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.135866    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-oauth-config" pod "62a79f63-2cb9-4102-9abf-6d431267b1bf" (UID: "62a79f63-2cb9-4102-9abf-6d431267b1bf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:20:06.135894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.135881    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-serving-cert" pod "62a79f63-2cb9-4102-9abf-6d431267b1bf" (UID: "62a79f63-2cb9-4102-9abf-6d431267b1bf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:20:06.136027 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.135915    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a79f63-2cb9-4102-9abf-6d431267b1bf-kube-api-access-5864h" pod "62a79f63-2cb9-4102-9abf-6d431267b1bf" (UID: "62a79f63-2cb9-4102-9abf-6d431267b1bf"). InnerVolumeSpecName "kube-api-access-5864h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:20:06.234804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.234768    4555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-serving-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:06.234804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.234798    4555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-config\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:06.234804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.234807    4555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-oauth-serving-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:06.235044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.234817    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5864h\" (UniqueName: \"kubernetes.io/projected/62a79f63-2cb9-4102-9abf-6d431267b1bf-kube-api-access-5864h\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:06.235044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.234828    4555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62a79f63-2cb9-4102-9abf-6d431267b1bf-console-oauth-config\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:06.235044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.234836    4555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-trusted-ca-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:06.235044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.234845    4555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62a79f63-2cb9-4102-9abf-6d431267b1bf-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:06.550847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.550815    4555 generic.go:356] "Generic (PLEG): container finished" podID="62a79f63-2cb9-4102-9abf-6d431267b1bf" containerID="6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9" exitCode=0
Oct 30 23:20:06.551032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.550891    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b898fd4b9-9mvfg"
Oct 30 23:20:06.551032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.550923    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b898fd4b9-9mvfg" event={"ID":"62a79f63-2cb9-4102-9abf-6d431267b1bf","Type":"ContainerDied","Data":"6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9"}
Oct 30 23:20:06.551032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.550982    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b898fd4b9-9mvfg" event={"ID":"62a79f63-2cb9-4102-9abf-6d431267b1bf","Type":"ContainerDied","Data":"8682937ce9e37f3dcf1cee1cd94adf59ac5c8f0729674b374ca13f16663f2a60"}
Oct 30 23:20:06.551032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.551004    4555 scope.go:122] "RemoveContainer" containerID="6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9"
Oct 30 23:20:06.559588 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.559573    4555 scope.go:122] "RemoveContainer" containerID="6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9"
Oct 30 23:20:06.559844 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:20:06.559829    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9\": container with ID starting with 6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9 not found: ID does not exist" containerID="6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9"
Oct 30 23:20:06.559894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.559855    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9"} err="failed to get container status \"6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9\": rpc error: code = NotFound desc = could not find container \"6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9\": container with ID starting with 6364b04ee03660b5051a9674dec489e800917f68f5616559522111aa5baf76a9 not found: ID does not exist"
Oct 30 23:20:06.572089 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.572066    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b898fd4b9-9mvfg"]
Oct 30 23:20:06.576397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:06.576353    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b898fd4b9-9mvfg"]
Oct 30 23:20:07.204265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:07.204233    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a79f63-2cb9-4102-9abf-6d431267b1bf" path="/var/lib/kubelet/pods/62a79f63-2cb9-4102-9abf-6d431267b1bf/volumes"
Oct 30 23:20:13.089996 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:13.089956    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:13.090365 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:13.090020    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:13.095045 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:13.095023    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:13.580278 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:13.580250    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:20:13.632465 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:13.632426    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b97c77649-8g86l"]
Oct 30 23:20:16.579753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.579699    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8589549b96-ztxc9" podUID="ec142d60-8f90-44f0-a57f-4de2294a028c" containerName="console" containerID="cri-o://163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad" gracePeriod=15
Oct 30 23:20:16.858422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.858344    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:20:16.923414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.923352    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-oauth-config\") pod \"ec142d60-8f90-44f0-a57f-4de2294a028c\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") "
Oct 30 23:20:16.923589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.923451    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-serving-cert\") pod \"ec142d60-8f90-44f0-a57f-4de2294a028c\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") "
Oct 30 23:20:16.923589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.923479    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-console-config\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-console-config\") pod \"ec142d60-8f90-44f0-a57f-4de2294a028c\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") "
Oct 30 23:20:16.923589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.923514    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec142d60-8f90-44f0-a57f-4de2294a028c-kube-api-access-2jw2g\" (UniqueName: \"kubernetes.io/projected/ec142d60-8f90-44f0-a57f-4de2294a028c-kube-api-access-2jw2g\") pod \"ec142d60-8f90-44f0-a57f-4de2294a028c\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") "
Oct 30 23:20:16.923589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.923539    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec142d60-8f90-44f0-a57f-4de2294a028c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec142d60-8f90-44f0-a57f-4de2294a028c-tmp\") pod \"ec142d60-8f90-44f0-a57f-4de2294a028c\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") "
Oct 30 23:20:16.923589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.923580    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-oauth-serving-cert\") pod \"ec142d60-8f90-44f0-a57f-4de2294a028c\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") "
Oct 30 23:20:16.923856 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.923613    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-service-ca\") pod \"ec142d60-8f90-44f0-a57f-4de2294a028c\" (UID: \"ec142d60-8f90-44f0-a57f-4de2294a028c\") "
Oct 30 23:20:16.924013 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.923981    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec142d60-8f90-44f0-a57f-4de2294a028c-tmp" pod "ec142d60-8f90-44f0-a57f-4de2294a028c" (UID: "ec142d60-8f90-44f0-a57f-4de2294a028c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:20:16.924185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.923981    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-console-config" pod "ec142d60-8f90-44f0-a57f-4de2294a028c" (UID: "ec142d60-8f90-44f0-a57f-4de2294a028c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:16.924185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.924072    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-oauth-serving-cert" pod "ec142d60-8f90-44f0-a57f-4de2294a028c" (UID: "ec142d60-8f90-44f0-a57f-4de2294a028c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:16.924185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.924083    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-service-ca" pod "ec142d60-8f90-44f0-a57f-4de2294a028c" (UID: "ec142d60-8f90-44f0-a57f-4de2294a028c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:16.925748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.925727    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec142d60-8f90-44f0-a57f-4de2294a028c-kube-api-access-2jw2g" pod "ec142d60-8f90-44f0-a57f-4de2294a028c" (UID: "ec142d60-8f90-44f0-a57f-4de2294a028c"). InnerVolumeSpecName "kube-api-access-2jw2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:20:16.926196 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.926171    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-oauth-config" pod "ec142d60-8f90-44f0-a57f-4de2294a028c" (UID: "ec142d60-8f90-44f0-a57f-4de2294a028c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:20:16.926196 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:16.926184    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-serving-cert" pod "ec142d60-8f90-44f0-a57f-4de2294a028c" (UID: "ec142d60-8f90-44f0-a57f-4de2294a028c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:20:17.024779 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.024745    4555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-oauth-serving-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:17.024779 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.024775    4555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:17.024779 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.024784    4555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-oauth-config\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:17.025014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.024793    4555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec142d60-8f90-44f0-a57f-4de2294a028c-console-serving-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:17.025014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.024802    4555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec142d60-8f90-44f0-a57f-4de2294a028c-console-config\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:17.025014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.024811    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jw2g\" (UniqueName: \"kubernetes.io/projected/ec142d60-8f90-44f0-a57f-4de2294a028c-kube-api-access-2jw2g\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:17.025014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.024820    4555 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec142d60-8f90-44f0-a57f-4de2294a028c-tmp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:17.590581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.590550    4555 generic.go:356] "Generic (PLEG): container finished" podID="ec142d60-8f90-44f0-a57f-4de2294a028c" containerID="163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad" exitCode=0
Oct 30 23:20:17.591028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.590595    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8589549b96-ztxc9"
Oct 30 23:20:17.591028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.590599    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8589549b96-ztxc9" event={"ID":"ec142d60-8f90-44f0-a57f-4de2294a028c","Type":"ContainerDied","Data":"163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad"}
Oct 30 23:20:17.591028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.590699    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8589549b96-ztxc9" event={"ID":"ec142d60-8f90-44f0-a57f-4de2294a028c","Type":"ContainerDied","Data":"704049d00f8a431b9807f43ebaaf452b5f64a4ff33dcc581f87172dd4bd038dc"}
Oct 30 23:20:17.591028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.590718    4555 scope.go:122] "RemoveContainer" containerID="163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad"
Oct 30 23:20:17.599170 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.599150    4555 scope.go:122] "RemoveContainer" containerID="163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad"
Oct 30 23:20:17.599483 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:20:17.599460    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad\": container with ID starting with 163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad not found: ID does not exist" containerID="163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad"
Oct 30 23:20:17.599559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.599496    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad"} err="failed to get container status \"163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad\": rpc error: code = NotFound desc = could not find container \"163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad\": container with ID starting with 163f20fde80b690ecd586f3bf088521e95712e29ff4b1654c2a1ffc3d63002ad not found: ID does not exist"
Oct 30 23:20:17.606740 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.606714    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8589549b96-ztxc9"]
Oct 30 23:20:17.611003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:17.610983    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8589549b96-ztxc9"]
Oct 30 23:20:19.207256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:19.207220    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec142d60-8f90-44f0-a57f-4de2294a028c" path="/var/lib/kubelet/pods/ec142d60-8f90-44f0-a57f-4de2294a028c/volumes"
Oct 30 23:20:27.444564 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:20:27.444524    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:20:29.207633 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.207602    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-lcxv4-ckdck"]
Oct 30 23:20:29.208048 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.207763    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-lcxv4-ckdck" podUID="2d37f12b-4177-44dd-8d90-a38eca561963" containerName="insights-gathering" containerID="cri-o://c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c" gracePeriod=30
Oct 30 23:20:29.459087 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.459022    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:20:29.626004 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.625968    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d37f12b-4177-44dd-8d90-a38eca561963-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d37f12b-4177-44dd-8d90-a38eca561963-archives-path\") pod \"2d37f12b-4177-44dd-8d90-a38eca561963\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") "
Oct 30 23:20:29.626219 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.626019    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-trusted-ca-bundle\") pod \"2d37f12b-4177-44dd-8d90-a38eca561963\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") "
Oct 30 23:20:29.626219 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.626056    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-service-ca-bundle\") pod \"2d37f12b-4177-44dd-8d90-a38eca561963\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") "
Oct 30 23:20:29.626219 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.626104    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d37f12b-4177-44dd-8d90-a38eca561963-kube-api-access-fnbkw\" (UniqueName: \"kubernetes.io/projected/2d37f12b-4177-44dd-8d90-a38eca561963-kube-api-access-fnbkw\") pod \"2d37f12b-4177-44dd-8d90-a38eca561963\" (UID: \"2d37f12b-4177-44dd-8d90-a38eca561963\") "
Oct 30 23:20:29.626468 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.626435    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d37f12b-4177-44dd-8d90-a38eca561963-archives-path" pod "2d37f12b-4177-44dd-8d90-a38eca561963" (UID: "2d37f12b-4177-44dd-8d90-a38eca561963"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:20:29.626546 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.626450    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-service-ca-bundle" pod "2d37f12b-4177-44dd-8d90-a38eca561963" (UID: "2d37f12b-4177-44dd-8d90-a38eca561963"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:29.626546 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.626511    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-trusted-ca-bundle" pod "2d37f12b-4177-44dd-8d90-a38eca561963" (UID: "2d37f12b-4177-44dd-8d90-a38eca561963"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:29.628413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.628388    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d37f12b-4177-44dd-8d90-a38eca561963-kube-api-access-fnbkw" pod "2d37f12b-4177-44dd-8d90-a38eca561963" (UID: "2d37f12b-4177-44dd-8d90-a38eca561963"). InnerVolumeSpecName "kube-api-access-fnbkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:20:29.633597 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.633563    4555 generic.go:356] "Generic (PLEG): container finished" podID="2d37f12b-4177-44dd-8d90-a38eca561963" containerID="c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c" exitCode=2
Oct 30 23:20:29.633694 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.633632    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lcxv4-ckdck"
Oct 30 23:20:29.633738 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.633631    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lcxv4-ckdck" event={"ID":"2d37f12b-4177-44dd-8d90-a38eca561963","Type":"ContainerDied","Data":"c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c"}
Oct 30 23:20:29.633776 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.633751    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lcxv4-ckdck" event={"ID":"2d37f12b-4177-44dd-8d90-a38eca561963","Type":"ContainerDied","Data":"65cea0b9e303b01548517c69a5ddfdd88aa6d5509593406a8bd8c4798ce7d0a8"}
Oct 30 23:20:29.633776 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.633770    4555 scope.go:122] "RemoveContainer" containerID="c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c"
Oct 30 23:20:29.641633 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.641609    4555 scope.go:122] "RemoveContainer" containerID="c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c"
Oct 30 23:20:29.642011 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:20:29.641980    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c\": container with ID starting with c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c not found: ID does not exist" containerID="c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c"
Oct 30 23:20:29.642125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.642012    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c"} err="failed to get container status \"c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c\": rpc error: code = NotFound desc = could not find container \"c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c\": container with ID starting with c9ec76a7a9e8e167970710ab0333a9b5ba57f981c619876ceab08df80d10ce0c not found: ID does not exist"
Oct 30 23:20:29.655436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.655409    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-lcxv4-ckdck"]
Oct 30 23:20:29.659563 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.659533    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-lcxv4-ckdck"]
Oct 30 23:20:29.727423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.727306    4555 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d37f12b-4177-44dd-8d90-a38eca561963-archives-path\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:29.727423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.727338    4555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-trusted-ca-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:29.727423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.727350    4555 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d37f12b-4177-44dd-8d90-a38eca561963-service-ca-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:29.727423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:29.727360    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnbkw\" (UniqueName: \"kubernetes.io/projected/2d37f12b-4177-44dd-8d90-a38eca561963-kube-api-access-fnbkw\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:31.204463 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:31.204428    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d37f12b-4177-44dd-8d90-a38eca561963" path="/var/lib/kubelet/pods/2d37f12b-4177-44dd-8d90-a38eca561963/volumes"
Oct 30 23:20:38.657510 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:38.657469    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6b97c77649-8g86l" podUID="6342fbca-d47c-4d4f-8e65-0fcade1a2107" containerName="console" containerID="cri-o://c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5" gracePeriod=15
Oct 30 23:20:38.923143 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:38.923118    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:20:39.092409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092331    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-config\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-config\") pod \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") "
Oct 30 23:20:39.092409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092425    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-oauth-serving-cert\") pod \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") "
Oct 30 23:20:39.092652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092451    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6342fbca-d47c-4d4f-8e65-0fcade1a2107-kube-api-access-mpblg\" (UniqueName: \"kubernetes.io/projected/6342fbca-d47c-4d4f-8e65-0fcade1a2107-kube-api-access-mpblg\") pod \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") "
Oct 30 23:20:39.092652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092495    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-service-ca\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-service-ca\") pod \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") "
Oct 30 23:20:39.092652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092533    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6342fbca-d47c-4d4f-8e65-0fcade1a2107-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6342fbca-d47c-4d4f-8e65-0fcade1a2107-tmp\") pod \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") "
Oct 30 23:20:39.092652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092564    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-oauth-config\") pod \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") "
Oct 30 23:20:39.092652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092609    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-trusted-ca-bundle\") pod \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") "
Oct 30 23:20:39.092868 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092690    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-serving-cert\") pod \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\" (UID: \"6342fbca-d47c-4d4f-8e65-0fcade1a2107\") "
Oct 30 23:20:39.092868 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092811    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-config" pod "6342fbca-d47c-4d4f-8e65-0fcade1a2107" (UID: "6342fbca-d47c-4d4f-8e65-0fcade1a2107"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:39.092976 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092920    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-oauth-serving-cert" pod "6342fbca-d47c-4d4f-8e65-0fcade1a2107" (UID: "6342fbca-d47c-4d4f-8e65-0fcade1a2107"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:39.093030 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.092991    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6342fbca-d47c-4d4f-8e65-0fcade1a2107-tmp" pod "6342fbca-d47c-4d4f-8e65-0fcade1a2107" (UID: "6342fbca-d47c-4d4f-8e65-0fcade1a2107"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:20:39.093096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.093071    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-service-ca" pod "6342fbca-d47c-4d4f-8e65-0fcade1a2107" (UID: "6342fbca-d47c-4d4f-8e65-0fcade1a2107"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:39.093096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.093087    4555 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6342fbca-d47c-4d4f-8e65-0fcade1a2107-tmp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:39.093199 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.093106    4555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-config\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:39.093199 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.093109    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-trusted-ca-bundle" pod "6342fbca-d47c-4d4f-8e65-0fcade1a2107" (UID: "6342fbca-d47c-4d4f-8e65-0fcade1a2107"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:39.093199 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.093120    4555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-oauth-serving-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:39.095031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.094991    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-oauth-config" pod "6342fbca-d47c-4d4f-8e65-0fcade1a2107" (UID: "6342fbca-d47c-4d4f-8e65-0fcade1a2107"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:20:39.095031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.095008    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6342fbca-d47c-4d4f-8e65-0fcade1a2107-kube-api-access-mpblg" pod "6342fbca-d47c-4d4f-8e65-0fcade1a2107" (UID: "6342fbca-d47c-4d4f-8e65-0fcade1a2107"). InnerVolumeSpecName "kube-api-access-mpblg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:20:39.095031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.095011    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-serving-cert" pod "6342fbca-d47c-4d4f-8e65-0fcade1a2107" (UID: "6342fbca-d47c-4d4f-8e65-0fcade1a2107"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:20:39.194454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.194348    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpblg\" (UniqueName: \"kubernetes.io/projected/6342fbca-d47c-4d4f-8e65-0fcade1a2107-kube-api-access-mpblg\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:39.194454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.194402    4555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:39.194454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.194412    4555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-oauth-config\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:39.194454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.194422    4555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6342fbca-d47c-4d4f-8e65-0fcade1a2107-trusted-ca-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:39.194454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.194430    4555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6342fbca-d47c-4d4f-8e65-0fcade1a2107-console-serving-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:39.672350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.672315    4555 generic.go:356] "Generic (PLEG): container finished" podID="6342fbca-d47c-4d4f-8e65-0fcade1a2107" containerID="c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5" exitCode=0
Oct 30 23:20:39.672787 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.672380    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b97c77649-8g86l"
Oct 30 23:20:39.672787 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.672405    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b97c77649-8g86l" event={"ID":"6342fbca-d47c-4d4f-8e65-0fcade1a2107","Type":"ContainerDied","Data":"c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5"}
Oct 30 23:20:39.672787 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.672462    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b97c77649-8g86l" event={"ID":"6342fbca-d47c-4d4f-8e65-0fcade1a2107","Type":"ContainerDied","Data":"fa7ba13bb4659b3b702fbbaf1c2385e530059ddd1663844dba17342b9542ff4b"}
Oct 30 23:20:39.672787 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.672489    4555 scope.go:122] "RemoveContainer" containerID="c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5"
Oct 30 23:20:39.680511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.680492    4555 scope.go:122] "RemoveContainer" containerID="c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5"
Oct 30 23:20:39.680834 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:20:39.680814    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5\": container with ID starting with c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5 not found: ID does not exist" containerID="c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5"
Oct 30 23:20:39.680916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.680843    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5"} err="failed to get container status \"c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5\": rpc error: code = NotFound desc = could not find container \"c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5\": container with ID starting with c0eeb178dd4fede9a38c1309c7f6b989a00ab70797204adaea2542d06b399fd5 not found: ID does not exist"
Oct 30 23:20:39.690003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.689979    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b97c77649-8g86l"]
Oct 30 23:20:39.694209 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:39.694186    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b97c77649-8g86l"]
Oct 30 23:20:41.204696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:20:41.204662    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6342fbca-d47c-4d4f-8e65-0fcade1a2107" path="/var/lib/kubelet/pods/6342fbca-d47c-4d4f-8e65-0fcade1a2107/volumes"
Oct 30 23:20:57.445438 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:20:57.445400    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:21:27.445465 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:21:27.445382    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:21:46.365125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365089    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"]
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365336    4555 cpu_manager.go:414] "containerMap: removing container" podUID="2d37f12b-4177-44dd-8d90-a38eca561963" containerName="insights-gathering"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365347    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d37f12b-4177-44dd-8d90-a38eca561963" containerName="insights-gathering"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365358    4555 cpu_manager.go:414] "containerMap: removing container" podUID="ec142d60-8f90-44f0-a57f-4de2294a028c" containerName="console"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365363    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec142d60-8f90-44f0-a57f-4de2294a028c" containerName="console"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365386    4555 cpu_manager.go:414] "containerMap: removing container" podUID="62a79f63-2cb9-4102-9abf-6d431267b1bf" containerName="console"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365394    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a79f63-2cb9-4102-9abf-6d431267b1bf" containerName="console"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365410    4555 cpu_manager.go:414] "containerMap: removing container" podUID="6342fbca-d47c-4d4f-8e65-0fcade1a2107" containerName="console"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365415    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6342fbca-d47c-4d4f-8e65-0fcade1a2107" containerName="console"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365453    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d37f12b-4177-44dd-8d90-a38eca561963" containerName="insights-gathering"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365461    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec142d60-8f90-44f0-a57f-4de2294a028c" containerName="console"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365466    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a79f63-2cb9-4102-9abf-6d431267b1bf" containerName="console"
Oct 30 23:21:46.365620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.365473    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="6342fbca-d47c-4d4f-8e65-0fcade1a2107" containerName="console"
Oct 30 23:21:46.367290 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.367273    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.378914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.378891    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 23:21:46.379751 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.379728    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-ct586\""
Oct 30 23:21:46.379842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.379752    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 23:21:46.383806 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.383785    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"]
Oct 30 23:21:46.460795 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.460761    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.460954 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.460810    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.460954 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.460874    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fst8\" (UniqueName: \"kubernetes.io/projected/ab8e0839-7dfb-4255-9138-a041a06370cc-kube-api-access-7fst8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.562096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.562057    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fst8\" (UniqueName: \"kubernetes.io/projected/ab8e0839-7dfb-4255-9138-a041a06370cc-kube-api-access-7fst8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.562265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.562130    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.562265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.562179    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.562603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.562585    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.562641 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.562613    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.579802 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.579769    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fst8\" (UniqueName: \"kubernetes.io/projected/ab8e0839-7dfb-4255-9138-a041a06370cc-kube-api-access-7fst8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.608281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.608252    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"]
Oct 30 23:21:46.610284 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.610269    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.624108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.623292    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"]
Oct 30 23:21:46.662595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.662560    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.662771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.662624    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.662771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.662709    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwbhw\" (UniqueName: \"kubernetes.io/projected/40a92c96-b669-4712-8362-47f991b6555a-kube-api-access-pwbhw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.679524 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.679498    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:21:46.690773 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.690745    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-str7h\""
Oct 30 23:21:46.763923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.763884    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.764081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.763928    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwbhw\" (UniqueName: \"kubernetes.io/projected/40a92c96-b669-4712-8362-47f991b6555a-kube-api-access-pwbhw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.764081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.763966    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.764332 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.764310    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.764401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.764317    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.777750 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.777730    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwbhw\" (UniqueName: \"kubernetes.io/projected/40a92c96-b669-4712-8362-47f991b6555a-kube-api-access-pwbhw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.928600 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.928523    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:21:46.956942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:46.956913    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"]
Oct 30 23:21:46.960097 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:46.960071    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8e0839_7dfb_4255_9138_a041a06370cc.slice/crio-2671acae46fa66ccfcbda478a05aed2d9a16b7bd57e23c6868bff3120da982f0 WatchSource:0}: Error finding container 2671acae46fa66ccfcbda478a05aed2d9a16b7bd57e23c6868bff3120da982f0: Status 404 returned error can't find the container with id 2671acae46fa66ccfcbda478a05aed2d9a16b7bd57e23c6868bff3120da982f0
Oct 30 23:21:47.221936 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:47.221866    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"]
Oct 30 23:21:47.224552 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:47.224527    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a92c96_b669_4712_8362_47f991b6555a.slice/crio-e928a0db581545340ca376b2568500c23903f0f598a6774d393dd819c995e4fb WatchSource:0}: Error finding container e928a0db581545340ca376b2568500c23903f0f598a6774d393dd819c995e4fb: Status 404 returned error can't find the container with id e928a0db581545340ca376b2568500c23903f0f598a6774d393dd819c995e4fb
Oct 30 23:21:47.894545 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:47.894501    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df" event={"ID":"ab8e0839-7dfb-4255-9138-a041a06370cc","Type":"ContainerStarted","Data":"2671acae46fa66ccfcbda478a05aed2d9a16b7bd57e23c6868bff3120da982f0"}
Oct 30 23:21:47.896032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:47.895999    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c" event={"ID":"40a92c96-b669-4712-8362-47f991b6555a","Type":"ContainerStarted","Data":"e928a0db581545340ca376b2568500c23903f0f598a6774d393dd819c995e4fb"}
Oct 30 23:21:48.982921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:48.982884    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"]
Oct 30 23:21:48.985870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:48.985846    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:48.991156 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:48.991132    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-krnkz\""
Oct 30 23:21:48.996023 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:48.995997    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"]
Oct 30 23:21:49.083742 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.083703    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpzcp\" (UniqueName: \"kubernetes.io/projected/6f0e8cd6-4b50-4886-9616-750da4c045c9-kube-api-access-rpzcp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:49.083941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.083844    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:49.083941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.083883    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:49.185361 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.185316    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpzcp\" (UniqueName: \"kubernetes.io/projected/6f0e8cd6-4b50-4886-9616-750da4c045c9-kube-api-access-rpzcp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:49.185562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.185426    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:49.185562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.185458    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:49.185874 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.185852    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:49.186111 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.186086    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:49.199535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.199506    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpzcp\" (UniqueName: \"kubernetes.io/projected/6f0e8cd6-4b50-4886-9616-750da4c045c9-kube-api-access-rpzcp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:49.305579 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.305547    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:21:49.525309 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.525271    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"]
Oct 30 23:21:49.528002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.527702    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.532002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.531980    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-bl8rj\""
Oct 30 23:21:49.538989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.538968    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"]
Oct 30 23:21:49.589132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.589049    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmn2l\" (UniqueName: \"kubernetes.io/projected/118a6242-b02d-49fd-9d92-59e412aa1a5f-kube-api-access-wmn2l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.589132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.589120    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.589395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.589164    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.592140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.591323    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"]
Oct 30 23:21:49.596193 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:49.596166    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0e8cd6_4b50_4886_9616_750da4c045c9.slice/crio-1dad373ca0f438df6a2e9edef9b1a5164c42871337b225d9a96962925995d43a WatchSource:0}: Error finding container 1dad373ca0f438df6a2e9edef9b1a5164c42871337b225d9a96962925995d43a: Status 404 returned error can't find the container with id 1dad373ca0f438df6a2e9edef9b1a5164c42871337b225d9a96962925995d43a
Oct 30 23:21:49.624650 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.624615    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"]
Oct 30 23:21:49.627364 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.627340    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.632945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.632925    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-l2dnn\""
Oct 30 23:21:49.638397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.638344    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"]
Oct 30 23:21:49.690193 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.690157    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.690412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.690214    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.690412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.690252    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.690412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.690290    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.690412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.690322    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xncf6\" (UniqueName: \"kubernetes.io/projected/e73d1c0e-a5c4-4166-828a-49c191300b22-kube-api-access-xncf6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.690412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.690397    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmn2l\" (UniqueName: \"kubernetes.io/projected/118a6242-b02d-49fd-9d92-59e412aa1a5f-kube-api-access-wmn2l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.690679 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.690607    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.690847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.690825    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.706548 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.706511    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmn2l\" (UniqueName: \"kubernetes.io/projected/118a6242-b02d-49fd-9d92-59e412aa1a5f-kube-api-access-wmn2l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.725765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.725731    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"]
Oct 30 23:21:49.728947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.728924    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:49.738151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.737622    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-wcrfj\""
Oct 30 23:21:49.738285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.738209    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"]
Oct 30 23:21:49.791017 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.790976    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:49.791260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.791028    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.791260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.791067    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.791260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.791086    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtvw4\" (UniqueName: \"kubernetes.io/projected/7faba7c5-348f-4137-8e79-f220d806181c-kube-api-access-xtvw4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:49.791260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.791121    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xncf6\" (UniqueName: \"kubernetes.io/projected/e73d1c0e-a5c4-4166-828a-49c191300b22-kube-api-access-xncf6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.791260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.791167    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:49.791781 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.791564    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.791781 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.791756    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.809413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.809300    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xncf6\" (UniqueName: \"kubernetes.io/projected/e73d1c0e-a5c4-4166-828a-49c191300b22-kube-api-access-xncf6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.845637 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.845547    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:21:49.857174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.857146    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"]
Oct 30 23:21:49.859913 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.859891    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:49.864255 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.864236    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-nljbm\""
Oct 30 23:21:49.870160 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.869967    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"]
Oct 30 23:21:49.892939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.891775    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:49.892939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.891851    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtvw4\" (UniqueName: \"kubernetes.io/projected/7faba7c5-348f-4137-8e79-f220d806181c-kube-api-access-xtvw4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:49.892939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.891931    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:49.892939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.892394    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:49.892939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.892645    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:49.915039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.914456    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtvw4\" (UniqueName: \"kubernetes.io/projected/7faba7c5-348f-4137-8e79-f220d806181c-kube-api-access-xtvw4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:49.915566 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.915466    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6" event={"ID":"6f0e8cd6-4b50-4886-9616-750da4c045c9","Type":"ContainerStarted","Data":"1dad373ca0f438df6a2e9edef9b1a5164c42871337b225d9a96962925995d43a"}
Oct 30 23:21:49.956888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.956509    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:21:49.992875 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.992828    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7hm\" (UniqueName: \"kubernetes.io/projected/1f7362fd-b371-4e8b-a428-9e548ebae04c-kube-api-access-hn7hm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:49.993415 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.992973    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:49.993415 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:49.993020    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:50.047974 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.047939    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:21:50.094069 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.094033    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:50.094246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.094082    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:50.094246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.094132    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7hm\" (UniqueName: \"kubernetes.io/projected/1f7362fd-b371-4e8b-a428-9e548ebae04c-kube-api-access-hn7hm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:50.094806 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.094778    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:50.095023 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.095004    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:50.111811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.108260    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7hm\" (UniqueName: \"kubernetes.io/projected/1f7362fd-b371-4e8b-a428-9e548ebae04c-kube-api-access-hn7hm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:50.142336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.142304    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"]
Oct 30 23:21:50.153886 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:50.153860    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118a6242_b02d_49fd_9d92_59e412aa1a5f.slice/crio-9cddec2af070dd2ec319e019d3a9327276e6907d3bdd51fe86039c7c257a8398 WatchSource:0}: Error finding container 9cddec2af070dd2ec319e019d3a9327276e6907d3bdd51fe86039c7c257a8398: Status 404 returned error can't find the container with id 9cddec2af070dd2ec319e019d3a9327276e6907d3bdd51fe86039c7c257a8398
Oct 30 23:21:50.179680 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.179650    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:21:50.250350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.249358    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"]
Oct 30 23:21:50.263109 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:50.263055    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73d1c0e_a5c4_4166_828a_49c191300b22.slice/crio-1790328bc1addf2fa988ed8f9f6f187b48fa86e4934738ec289f865cca25f342 WatchSource:0}: Error finding container 1790328bc1addf2fa988ed8f9f6f187b48fa86e4934738ec289f865cca25f342: Status 404 returned error can't find the container with id 1790328bc1addf2fa988ed8f9f6f187b48fa86e4934738ec289f865cca25f342
Oct 30 23:21:50.331306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.331049    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"]
Oct 30 23:21:50.334612 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:50.334580    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7faba7c5_348f_4137_8e79_f220d806181c.slice/crio-5996cd93360aa399f0e45dec7c7755d4d7ed4a4f5e451d7a9637dd9ee3bc6f9d WatchSource:0}: Error finding container 5996cd93360aa399f0e45dec7c7755d4d7ed4a4f5e451d7a9637dd9ee3bc6f9d: Status 404 returned error can't find the container with id 5996cd93360aa399f0e45dec7c7755d4d7ed4a4f5e451d7a9637dd9ee3bc6f9d
Oct 30 23:21:50.466465 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.466326    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"]
Oct 30 23:21:50.471218 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:50.471186    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7362fd_b371_4e8b_a428_9e548ebae04c.slice/crio-57ca664fd5391cb34ae16d5fd1038fd774e88253e6497e2a9662dbf77c530ee9 WatchSource:0}: Error finding container 57ca664fd5391cb34ae16d5fd1038fd774e88253e6497e2a9662dbf77c530ee9: Status 404 returned error can't find the container with id 57ca664fd5391cb34ae16d5fd1038fd774e88253e6497e2a9662dbf77c530ee9
Oct 30 23:21:50.926634 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.926536    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm" event={"ID":"1f7362fd-b371-4e8b-a428-9e548ebae04c","Type":"ContainerStarted","Data":"57ca664fd5391cb34ae16d5fd1038fd774e88253e6497e2a9662dbf77c530ee9"}
Oct 30 23:21:50.929053 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.928955    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv" event={"ID":"7faba7c5-348f-4137-8e79-f220d806181c","Type":"ContainerStarted","Data":"5996cd93360aa399f0e45dec7c7755d4d7ed4a4f5e451d7a9637dd9ee3bc6f9d"}
Oct 30 23:21:50.932257 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.932182    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2" event={"ID":"e73d1c0e-a5c4-4166-828a-49c191300b22","Type":"ContainerStarted","Data":"1790328bc1addf2fa988ed8f9f6f187b48fa86e4934738ec289f865cca25f342"}
Oct 30 23:21:50.936322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:50.936259    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6" event={"ID":"118a6242-b02d-49fd-9d92-59e412aa1a5f","Type":"ContainerStarted","Data":"9cddec2af070dd2ec319e019d3a9327276e6907d3bdd51fe86039c7c257a8398"}
Oct 30 23:21:51.801208 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:51.801176    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"]
Oct 30 23:21:51.804442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:51.804416    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:51.819239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:51.817019    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"]
Oct 30 23:21:51.912983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:51.912933    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:51.913185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:51.913023    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffzg9\" (UniqueName: \"kubernetes.io/projected/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-kube-api-access-ffzg9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:51.913185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:51.913086    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:52.015451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.014041    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:52.015451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.014122    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffzg9\" (UniqueName: \"kubernetes.io/projected/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-kube-api-access-ffzg9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:52.015451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.014214    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:52.015451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.014728    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:52.015451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.014982    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:52.029270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.029186    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffzg9\" (UniqueName: \"kubernetes.io/projected/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-kube-api-access-ffzg9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:52.124932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.124620    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:21:52.435445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.434323    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"]
Oct 30 23:21:52.441047 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:52.441018    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17194046_7ad7_4ebf_b2ec_c53b70e5ba8e.slice/crio-a36b457af1c51c1c2ae2842d8e76a059b98934528e27174aa3774a73a4118246 WatchSource:0}: Error finding container a36b457af1c51c1c2ae2842d8e76a059b98934528e27174aa3774a73a4118246: Status 404 returned error can't find the container with id a36b457af1c51c1c2ae2842d8e76a059b98934528e27174aa3774a73a4118246
Oct 30 23:21:52.826481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.826447    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"]
Oct 30 23:21:52.832763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.832463    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:52.846434 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.846398    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 23:21:52.846593 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.846495    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 23:21:52.846659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.846601    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 23:21:52.849998 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.849890    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"]
Oct 30 23:21:52.853431 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.853403    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rb9gl\""
Oct 30 23:21:52.926140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.926094    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea819a48-154f-4c7a-a747-3306ac423289-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:52.926396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.926157    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:52.926396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.926190    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:52.926396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.926216    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea819a48-154f-4c7a-a747-3306ac423289-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:52.926396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.926244    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:52.926396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.926302    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:52.926396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.926333    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:52.926396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.926397    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tcm5\" (UniqueName: \"kubernetes.io/projected/ea819a48-154f-4c7a-a747-3306ac423289-kube-api-access-6tcm5\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:52.926841 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.926427    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea819a48-154f-4c7a-a747-3306ac423289-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:52.952569 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.952534    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"]
Oct 30 23:21:52.963739 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.963704    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr" event={"ID":"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e","Type":"ContainerStarted","Data":"a36b457af1c51c1c2ae2842d8e76a059b98934528e27174aa3774a73a4118246"}
Oct 30 23:21:52.963898 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.963816    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:52.980749 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.980717    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-7hgcz\""
Oct 30 23:21:52.981670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:52.981650    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"]
Oct 30 23:21:53.015459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.014850    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"]
Oct 30 23:21:53.018909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.018883    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.026888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.026855    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tcm5\" (UniqueName: \"kubernetes.io/projected/ea819a48-154f-4c7a-a747-3306ac423289-kube-api-access-6tcm5\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.027012 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.026897    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea819a48-154f-4c7a-a747-3306ac423289-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.027012 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.026961    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea819a48-154f-4c7a-a747-3306ac423289-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.027012 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.026990    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.027189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.027019    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.027189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.027041    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea819a48-154f-4c7a-a747-3306ac423289-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.027189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.027069    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.027189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.027121    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.027189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.027148    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.027619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.027530    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.030462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.030180    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.031059 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.030993    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea819a48-154f-4c7a-a747-3306ac423289-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.031457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.031253    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.031457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.031398    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.037243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.037178    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"]
Oct 30 23:21:53.037631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.037603    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea819a48-154f-4c7a-a747-3306ac423289-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.040137 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.040115    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea819a48-154f-4c7a-a747-3306ac423289-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.064578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.064523    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tcm5\" (UniqueName: \"kubernetes.io/projected/ea819a48-154f-4c7a-a747-3306ac423289-kube-api-access-6tcm5\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.073396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.073330    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea819a48-154f-4c7a-a747-3306ac423289-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-mjk4b\" (UID: \"ea819a48-154f-4c7a-a747-3306ac423289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.128462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128356    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.128462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128431    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.128694 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128552    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.128694 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128612    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.128694 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128643    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.128694 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128673    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.128917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128725    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.128917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128758    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f308b264-593d-49b7-b1b9-03c09e9cee62-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.128917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128795    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.128917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128838    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf92p\" (UniqueName: \"kubernetes.io/projected/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-kube-api-access-qf92p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.128917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128871    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.128917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.128896    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsbnb\" (UniqueName: \"kubernetes.io/projected/f308b264-593d-49b7-b1b9-03c09e9cee62-kube-api-access-xsbnb\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.163926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.163652    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:21:53.230402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230344    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf92p\" (UniqueName: \"kubernetes.io/projected/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-kube-api-access-qf92p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.230582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230426    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.230582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230453    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsbnb\" (UniqueName: \"kubernetes.io/projected/f308b264-593d-49b7-b1b9-03c09e9cee62-kube-api-access-xsbnb\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.230582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230479    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.230582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230510    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.230582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230543    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.230582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230579    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.230893 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230603    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.230893 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230638    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.230893 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230692    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.230893 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230733    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f308b264-593d-49b7-b1b9-03c09e9cee62-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.230893 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.230777    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.231253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.231225    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.231860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.231835    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.233577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.232750    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.233577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.232771    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.233577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.233233    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.233577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.233517    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.233888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.233649    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f308b264-593d-49b7-b1b9-03c09e9cee62-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.237061 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.237035    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.238456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.238359    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.246228 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.246174    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f308b264-593d-49b7-b1b9-03c09e9cee62-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.247619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.247591    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf92p\" (UniqueName: \"kubernetes.io/projected/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-kube-api-access-qf92p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.251513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.251487    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsbnb\" (UniqueName: \"kubernetes.io/projected/f308b264-593d-49b7-b1b9-03c09e9cee62-kube-api-access-xsbnb\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78\" (UID: \"f308b264-593d-49b7-b1b9-03c09e9cee62\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.314213 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.313848    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:21:53.377401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.377091    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:21:53.512299 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.512265    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"]
Oct 30 23:21:53.520186 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:53.520151    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea819a48_154f_4c7a_a747_3306ac423289.slice/crio-45e9e8961d627db50b863bcf7554cdfc675a4e73130c5e4f91c9301d32cadc8a WatchSource:0}: Error finding container 45e9e8961d627db50b863bcf7554cdfc675a4e73130c5e4f91c9301d32cadc8a: Status 404 returned error can't find the container with id 45e9e8961d627db50b863bcf7554cdfc675a4e73130c5e4f91c9301d32cadc8a
Oct 30 23:21:53.642236 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.642108    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"]
Oct 30 23:21:53.647957 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:53.647924    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf308b264_593d_49b7_b1b9_03c09e9cee62.slice/crio-457a8c9ed3b8bda3a7e682e6c690d08f551b27cf1c2da1c3c148234eab7e3637 WatchSource:0}: Error finding container 457a8c9ed3b8bda3a7e682e6c690d08f551b27cf1c2da1c3c148234eab7e3637: Status 404 returned error can't find the container with id 457a8c9ed3b8bda3a7e682e6c690d08f551b27cf1c2da1c3c148234eab7e3637
Oct 30 23:21:53.715349 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.715304    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"]
Oct 30 23:21:53.718301 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:21:53.718268    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39c702b_05ee_40cb_8442_5cbc6ef62d1e.slice/crio-e7113752e442971a047fa984ff4b856eca8593c1a9f3da865c7c65453a5bcfe5 WatchSource:0}: Error finding container e7113752e442971a047fa984ff4b856eca8593c1a9f3da865c7c65453a5bcfe5: Status 404 returned error can't find the container with id e7113752e442971a047fa984ff4b856eca8593c1a9f3da865c7c65453a5bcfe5
Oct 30 23:21:53.974158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.974089    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78" event={"ID":"f308b264-593d-49b7-b1b9-03c09e9cee62","Type":"ContainerStarted","Data":"457a8c9ed3b8bda3a7e682e6c690d08f551b27cf1c2da1c3c148234eab7e3637"}
Oct 30 23:21:53.975934 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.975905    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" event={"ID":"ea819a48-154f-4c7a-a747-3306ac423289","Type":"ContainerStarted","Data":"45e9e8961d627db50b863bcf7554cdfc675a4e73130c5e4f91c9301d32cadc8a"}
Oct 30 23:21:53.979176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:21:53.979143    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg" event={"ID":"b39c702b-05ee-40cb-8442-5cbc6ef62d1e","Type":"ContainerStarted","Data":"e7113752e442971a047fa984ff4b856eca8593c1a9f3da865c7c65453a5bcfe5"}
Oct 30 23:21:57.534095 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:21:57.534050    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:22:03.103929 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:03.103842    4555 generic.go:356] "Generic (PLEG): container finished" podID="ab8e0839-7dfb-4255-9138-a041a06370cc" containerID="e2717cebd1a1381bf9a37531446b7d1a4c2e8914710b84ec07790b5dbb605b9f" exitCode=0
Oct 30 23:22:03.104495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:03.103955    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df" event={"ID":"ab8e0839-7dfb-4255-9138-a041a06370cc","Type":"ContainerDied","Data":"e2717cebd1a1381bf9a37531446b7d1a4c2e8914710b84ec07790b5dbb605b9f"}
Oct 30 23:22:03.971456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:03.970747    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:22:03.971456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:03.970837    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:22:03.971456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:03.970878    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:22:03.983701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:03.983057    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:22:03.983701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:03.983135    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:22:03.983701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:03.983176    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:22:04.119985 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:04.117951    4555 generic.go:356] "Generic (PLEG): container finished" podID="40a92c96-b669-4712-8362-47f991b6555a" containerID="b1f48a677f4783ffc10c55fea594618b1ea4bf5e6530dcb394a19fe9d4b6f932" exitCode=0
Oct 30 23:22:04.119985 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:04.118094    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c" event={"ID":"40a92c96-b669-4712-8362-47f991b6555a","Type":"ContainerDied","Data":"b1f48a677f4783ffc10c55fea594618b1ea4bf5e6530dcb394a19fe9d4b6f932"}
Oct 30 23:22:05.128324 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:05.128285    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78" event={"ID":"f308b264-593d-49b7-b1b9-03c09e9cee62","Type":"ContainerStarted","Data":"76c2e3001edbe8aa0012d25193b1f7712325232a66fa844cf96094c9a445ee67"}
Oct 30 23:22:05.131950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:05.131778    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr" event={"ID":"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e","Type":"ContainerStarted","Data":"652ccb62e69ae5f75ad3e515bb0742c3f708be6f8cf97a1b7f201e8b17bf2be6"}
Oct 30 23:22:05.135361 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:05.135312    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" event={"ID":"ea819a48-154f-4c7a-a747-3306ac423289","Type":"ContainerStarted","Data":"354d7b69d7bbf5ada5a45d5aff2724c410025e092dc2e590ff84b790d9036a03"}
Oct 30 23:22:05.155757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:05.155543    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78" podStartSLOduration=2.837848762 podStartE2EDuration="13.155530477s" totalImagesPullingTime="10.317681715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:05.154298806 +0000 UTC m=+184.080936399" watchObservedRunningTime="2026-10-30 23:22:05.155530477 +0000 UTC m=+184.082168057"
Oct 30 23:22:05.164865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:05.164836    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:22:05.172801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:05.172776    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:22:05.178295 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:05.178065    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" podStartSLOduration=2.724254206 podStartE2EDuration="13.178052743s" totalImagesPullingTime="10.453798537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:05.176790811 +0000 UTC m=+184.103428392" watchObservedRunningTime="2026-10-30 23:22:05.178052743 +0000 UTC m=+184.104690317"
Oct 30 23:22:05.317360 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:05.314449    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:22:05.320891 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:05.320641    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:22:06.154441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:06.154232    4555 generic.go:356] "Generic (PLEG): container finished" podID="40a92c96-b669-4712-8362-47f991b6555a" containerID="7ade5eaebbf2f2907e7e4c31f4aa856b3fafbef7cef0b22a28375d92726f25bf" exitCode=0
Oct 30 23:22:06.154896 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:06.154622    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c" event={"ID":"40a92c96-b669-4712-8362-47f991b6555a","Type":"ContainerDied","Data":"7ade5eaebbf2f2907e7e4c31f4aa856b3fafbef7cef0b22a28375d92726f25bf"}
Oct 30 23:22:06.157505 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:06.157396    4555 generic.go:356] "Generic (PLEG): container finished" podID="b39c702b-05ee-40cb-8442-5cbc6ef62d1e" containerID="c2399e4045ebd4422f36d6160d98072392bf7e91efa056c45c1d6373aca1a58e" exitCode=0
Oct 30 23:22:06.157583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:06.157552    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg" event={"ID":"b39c702b-05ee-40cb-8442-5cbc6ef62d1e","Type":"ContainerDied","Data":"c2399e4045ebd4422f36d6160d98072392bf7e91efa056c45c1d6373aca1a58e"}
Oct 30 23:22:06.164076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:06.161837    4555 generic.go:356] "Generic (PLEG): container finished" podID="17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" containerID="652ccb62e69ae5f75ad3e515bb0742c3f708be6f8cf97a1b7f201e8b17bf2be6" exitCode=0
Oct 30 23:22:06.164076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:06.163140    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr" event={"ID":"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e","Type":"ContainerDied","Data":"652ccb62e69ae5f75ad3e515bb0742c3f708be6f8cf97a1b7f201e8b17bf2be6"}
Oct 30 23:22:06.164076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:06.163187    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:22:06.164076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:06.163936    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:22:06.166670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:06.166585    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78"
Oct 30 23:22:06.166915 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:06.166896    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:22:07.173553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:07.173504    4555 generic.go:356] "Generic (PLEG): container finished" podID="b39c702b-05ee-40cb-8442-5cbc6ef62d1e" containerID="dd0b1512623ad89898f967c39e59e8a97778bb4461ae6f00be87f191f42eec5e" exitCode=0
Oct 30 23:22:07.174860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:07.174835    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg" event={"ID":"b39c702b-05ee-40cb-8442-5cbc6ef62d1e","Type":"ContainerDied","Data":"dd0b1512623ad89898f967c39e59e8a97778bb4461ae6f00be87f191f42eec5e"}
Oct 30 23:22:08.182102 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:08.182062    4555 generic.go:356] "Generic (PLEG): container finished" podID="6f0e8cd6-4b50-4886-9616-750da4c045c9" containerID="fbf2eededdbced0351db006452a61542a273f230d88d4b3e70d74ea1af47777d" exitCode=0
Oct 30 23:22:08.183429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:08.183400    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6" event={"ID":"6f0e8cd6-4b50-4886-9616-750da4c045c9","Type":"ContainerDied","Data":"fbf2eededdbced0351db006452a61542a273f230d88d4b3e70d74ea1af47777d"}
Oct 30 23:22:09.189788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:09.189742    4555 generic.go:356] "Generic (PLEG): container finished" podID="1f7362fd-b371-4e8b-a428-9e548ebae04c" containerID="c6a32dafe484426b7f0c767c98cf98d31e01dd2c0bb60da787e157858d28a91d" exitCode=0
Oct 30 23:22:09.190317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:09.189837    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm" event={"ID":"1f7362fd-b371-4e8b-a428-9e548ebae04c","Type":"ContainerDied","Data":"c6a32dafe484426b7f0c767c98cf98d31e01dd2c0bb60da787e157858d28a91d"}
Oct 30 23:22:09.193062 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:09.193031    4555 generic.go:356] "Generic (PLEG): container finished" podID="7faba7c5-348f-4137-8e79-f220d806181c" containerID="5677ad3c9021a391bae5e9f63a95e8ca7cf2e9be16ff3b7802182f8771ac3c6e" exitCode=0
Oct 30 23:22:09.193194 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:09.193143    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv" event={"ID":"7faba7c5-348f-4137-8e79-f220d806181c","Type":"ContainerDied","Data":"5677ad3c9021a391bae5e9f63a95e8ca7cf2e9be16ff3b7802182f8771ac3c6e"}
Oct 30 23:22:09.195843 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:09.195817    4555 generic.go:356] "Generic (PLEG): container finished" podID="e73d1c0e-a5c4-4166-828a-49c191300b22" containerID="4228cff126471f999b88522aee7f396f4327580569275849d4034404f01b3ed0" exitCode=0
Oct 30 23:22:09.195943 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:09.195893    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2" event={"ID":"e73d1c0e-a5c4-4166-828a-49c191300b22","Type":"ContainerDied","Data":"4228cff126471f999b88522aee7f396f4327580569275849d4034404f01b3ed0"}
Oct 30 23:22:09.198318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:09.198268    4555 generic.go:356] "Generic (PLEG): container finished" podID="118a6242-b02d-49fd-9d92-59e412aa1a5f" containerID="e6cf6993942ce3f7cb139cd97934ee59607f3c6fa23db5570560fd18656c0ea3" exitCode=0
Oct 30 23:22:09.198439 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:09.198405    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6" event={"ID":"118a6242-b02d-49fd-9d92-59e412aa1a5f","Type":"ContainerDied","Data":"e6cf6993942ce3f7cb139cd97934ee59607f3c6fa23db5570560fd18656c0ea3"}
Oct 30 23:22:10.208876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:10.208839    4555 generic.go:356] "Generic (PLEG): container finished" podID="ab8e0839-7dfb-4255-9138-a041a06370cc" containerID="5f350a71729595b5d6397ed6f316fa145f981816c81ccb2426e77f169652938a" exitCode=0
Oct 30 23:22:10.209311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:10.208970    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df" event={"ID":"ab8e0839-7dfb-4255-9138-a041a06370cc","Type":"ContainerDied","Data":"5f350a71729595b5d6397ed6f316fa145f981816c81ccb2426e77f169652938a"}
Oct 30 23:22:10.212265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:10.212236    4555 generic.go:356] "Generic (PLEG): container finished" podID="17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" containerID="8a8e823f8a1fff82f1a370ce0e46d26b684c90eba12a2b8a3548b2f7be470947" exitCode=0
Oct 30 23:22:10.212440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:10.212361    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr" event={"ID":"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e","Type":"ContainerDied","Data":"8a8e823f8a1fff82f1a370ce0e46d26b684c90eba12a2b8a3548b2f7be470947"}
Oct 30 23:22:10.217948 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:10.217837    4555 generic.go:356] "Generic (PLEG): container finished" podID="1f7362fd-b371-4e8b-a428-9e548ebae04c" containerID="25105c1242380879743b8c9910de321dd90f42b1b2b90c1de89073ae094ac1f0" exitCode=0
Oct 30 23:22:10.217948 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:10.217901    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm" event={"ID":"1f7362fd-b371-4e8b-a428-9e548ebae04c","Type":"ContainerDied","Data":"25105c1242380879743b8c9910de321dd90f42b1b2b90c1de89073ae094ac1f0"}
Oct 30 23:22:11.239151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:11.238000    4555 generic.go:356] "Generic (PLEG): container finished" podID="7faba7c5-348f-4137-8e79-f220d806181c" containerID="2af91653bdc57a12dba53b0dc4c1b9f6f074927d6fd9b2346fdecfe34a3d8a31" exitCode=0
Oct 30 23:22:11.239151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:11.238126    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv" event={"ID":"7faba7c5-348f-4137-8e79-f220d806181c","Type":"ContainerDied","Data":"2af91653bdc57a12dba53b0dc4c1b9f6f074927d6fd9b2346fdecfe34a3d8a31"}
Oct 30 23:22:11.245393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:11.245332    4555 generic.go:356] "Generic (PLEG): container finished" podID="e73d1c0e-a5c4-4166-828a-49c191300b22" containerID="de28e0185ea5b31df33eac6cdbebb6b54b3d17b4481bfaab61ba7111d7c5b3fc" exitCode=0
Oct 30 23:22:11.245679 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:11.245525    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2" event={"ID":"e73d1c0e-a5c4-4166-828a-49c191300b22","Type":"ContainerDied","Data":"de28e0185ea5b31df33eac6cdbebb6b54b3d17b4481bfaab61ba7111d7c5b3fc"}
Oct 30 23:22:11.254485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:11.254273    4555 generic.go:356] "Generic (PLEG): container finished" podID="118a6242-b02d-49fd-9d92-59e412aa1a5f" containerID="41e6431cc0f68a76426556b6b6dd70c9ffed70b0f1b66adefb80802a5032ed8a" exitCode=0
Oct 30 23:22:11.254485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:11.254316    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6" event={"ID":"118a6242-b02d-49fd-9d92-59e412aa1a5f","Type":"ContainerDied","Data":"41e6431cc0f68a76426556b6b6dd70c9ffed70b0f1b66adefb80802a5032ed8a"}
Oct 30 23:22:13.301966 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:13.301921    4555 generic.go:356] "Generic (PLEG): container finished" podID="6f0e8cd6-4b50-4886-9616-750da4c045c9" containerID="01ffbd269d3740eeacdb117a079b46a1689deb4c2f5c1e6c432a0259094c869c" exitCode=0
Oct 30 23:22:13.302481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:13.302013    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6" event={"ID":"6f0e8cd6-4b50-4886-9616-750da4c045c9","Type":"ContainerDied","Data":"01ffbd269d3740eeacdb117a079b46a1689deb4c2f5c1e6c432a0259094c869c"}
Oct 30 23:22:23.429110 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:23.426338    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv" event={"ID":"7faba7c5-348f-4137-8e79-f220d806181c","Type":"ContainerStarted","Data":"0bad9d6d78c5472d7b21a4872679623be51afe95b307729ffb4db6f0d66a9fcf"}
Oct 30 23:22:24.438390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:24.437899    4555 generic.go:356] "Generic (PLEG): container finished" podID="40a92c96-b669-4712-8362-47f991b6555a" containerID="9d94d6cb3eb48254cd5764bc6d5e86510f5bc0038b93051fdd7fa4e6c09d336b" exitCode=0
Oct 30 23:22:24.438390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:24.438009    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c" event={"ID":"40a92c96-b669-4712-8362-47f991b6555a","Type":"ContainerDied","Data":"9d94d6cb3eb48254cd5764bc6d5e86510f5bc0038b93051fdd7fa4e6c09d336b"}
Oct 30 23:22:24.443348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:24.443315    4555 generic.go:356] "Generic (PLEG): container finished" podID="7faba7c5-348f-4137-8e79-f220d806181c" containerID="0bad9d6d78c5472d7b21a4872679623be51afe95b307729ffb4db6f0d66a9fcf" exitCode=0
Oct 30 23:22:24.443493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:24.443469    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv" event={"ID":"7faba7c5-348f-4137-8e79-f220d806181c","Type":"ContainerDied","Data":"0bad9d6d78c5472d7b21a4872679623be51afe95b307729ffb4db6f0d66a9fcf"}
Oct 30 23:22:25.458459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.458055    4555 generic.go:356] "Generic (PLEG): container finished" podID="6f0e8cd6-4b50-4886-9616-750da4c045c9" containerID="f5e347296ed0b56264024ebc9f31c34249e002d9a72385d20bed3a1f6540fc0c" exitCode=0
Oct 30 23:22:25.458459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.458163    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6" event={"ID":"6f0e8cd6-4b50-4886-9616-750da4c045c9","Type":"ContainerDied","Data":"f5e347296ed0b56264024ebc9f31c34249e002d9a72385d20bed3a1f6540fc0c"}
Oct 30 23:22:25.463506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.463477    4555 generic.go:356] "Generic (PLEG): container finished" podID="b39c702b-05ee-40cb-8442-5cbc6ef62d1e" containerID="8c9969e1794612bac76af1c3cc5637d1ea62ad2ece4c11b5f3a1aeb282cd0cfc" exitCode=0
Oct 30 23:22:25.463892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.463868    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg" event={"ID":"b39c702b-05ee-40cb-8442-5cbc6ef62d1e","Type":"ContainerDied","Data":"8c9969e1794612bac76af1c3cc5637d1ea62ad2ece4c11b5f3a1aeb282cd0cfc"}
Oct 30 23:22:25.657615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.657576    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:22:25.722584 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.722514    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:22:25.776113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.775210    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-util\" (UniqueName: \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-util\") pod \"7faba7c5-348f-4137-8e79-f220d806181c\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") "
Oct 30 23:22:25.776113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.775323    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-bundle\") pod \"40a92c96-b669-4712-8362-47f991b6555a\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") "
Oct 30 23:22:25.776113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.775386    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-util\" (UniqueName: \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-util\") pod \"40a92c96-b669-4712-8362-47f991b6555a\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") "
Oct 30 23:22:25.776113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.775433    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7faba7c5-348f-4137-8e79-f220d806181c-kube-api-access-xtvw4\" (UniqueName: \"kubernetes.io/projected/7faba7c5-348f-4137-8e79-f220d806181c-kube-api-access-xtvw4\") pod \"7faba7c5-348f-4137-8e79-f220d806181c\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") "
Oct 30 23:22:25.776113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.775471    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a92c96-b669-4712-8362-47f991b6555a-kube-api-access-pwbhw\" (UniqueName: \"kubernetes.io/projected/40a92c96-b669-4712-8362-47f991b6555a-kube-api-access-pwbhw\") pod \"40a92c96-b669-4712-8362-47f991b6555a\" (UID: \"40a92c96-b669-4712-8362-47f991b6555a\") "
Oct 30 23:22:25.776113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.775532    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-bundle\") pod \"7faba7c5-348f-4137-8e79-f220d806181c\" (UID: \"7faba7c5-348f-4137-8e79-f220d806181c\") "
Oct 30 23:22:25.778469 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.776843    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-bundle" pod "40a92c96-b669-4712-8362-47f991b6555a" (UID: "40a92c96-b669-4712-8362-47f991b6555a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:25.779264 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.779215    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7faba7c5-348f-4137-8e79-f220d806181c-kube-api-access-xtvw4" pod "7faba7c5-348f-4137-8e79-f220d806181c" (UID: "7faba7c5-348f-4137-8e79-f220d806181c"). InnerVolumeSpecName "kube-api-access-xtvw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:25.779409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.779387    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-bundle" pod "7faba7c5-348f-4137-8e79-f220d806181c" (UID: "7faba7c5-348f-4137-8e79-f220d806181c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:25.781832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.781808    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a92c96-b669-4712-8362-47f991b6555a-kube-api-access-pwbhw" pod "40a92c96-b669-4712-8362-47f991b6555a" (UID: "40a92c96-b669-4712-8362-47f991b6555a"). InnerVolumeSpecName "kube-api-access-pwbhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:25.783391 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.783341    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-util" pod "7faba7c5-348f-4137-8e79-f220d806181c" (UID: "7faba7c5-348f-4137-8e79-f220d806181c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:25.785115 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.785092    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-util" pod "40a92c96-b669-4712-8362-47f991b6555a" (UID: "40a92c96-b669-4712-8362-47f991b6555a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:25.876641 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.876603    4555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-util\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:25.876641 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.876641    4555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:25.876872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.876653    4555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40a92c96-b669-4712-8362-47f991b6555a-util\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:25.876872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.876682    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtvw4\" (UniqueName: \"kubernetes.io/projected/7faba7c5-348f-4137-8e79-f220d806181c-kube-api-access-xtvw4\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:25.876872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.876694    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwbhw\" (UniqueName: \"kubernetes.io/projected/40a92c96-b669-4712-8362-47f991b6555a-kube-api-access-pwbhw\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:25.876872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:25.876707    4555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7faba7c5-348f-4137-8e79-f220d806181c-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:26.476162 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.476118    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c" event={"ID":"40a92c96-b669-4712-8362-47f991b6555a","Type":"ContainerDied","Data":"e928a0db581545340ca376b2568500c23903f0f598a6774d393dd819c995e4fb"}
Oct 30 23:22:26.476544 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.476187    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e928a0db581545340ca376b2568500c23903f0f598a6774d393dd819c995e4fb"
Oct 30 23:22:26.476544 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.476270    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qgf9c"
Oct 30 23:22:26.486615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.486584    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv"
Oct 30 23:22:26.488080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.487618    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avfqhv" event={"ID":"7faba7c5-348f-4137-8e79-f220d806181c","Type":"ContainerDied","Data":"5996cd93360aa399f0e45dec7c7755d4d7ed4a4f5e451d7a9637dd9ee3bc6f9d"}
Oct 30 23:22:26.488080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.487662    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5996cd93360aa399f0e45dec7c7755d4d7ed4a4f5e451d7a9637dd9ee3bc6f9d"
Oct 30 23:22:26.762074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.761995    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:22:26.774837 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.774805    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:22:26.884048 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.884011    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-bundle\") pod \"6f0e8cd6-4b50-4886-9616-750da4c045c9\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") "
Oct 30 23:22:26.884253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.884113    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-kube-api-access-qf92p\" (UniqueName: \"kubernetes.io/projected/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-kube-api-access-qf92p\") pod \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") "
Oct 30 23:22:26.884253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.884155    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-util\" (UniqueName: \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-util\") pod \"6f0e8cd6-4b50-4886-9616-750da4c045c9\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") "
Oct 30 23:22:26.884253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.884226    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-util\" (UniqueName: \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-util\") pod \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") "
Oct 30 23:22:26.884429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.884276    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0e8cd6-4b50-4886-9616-750da4c045c9-kube-api-access-rpzcp\" (UniqueName: \"kubernetes.io/projected/6f0e8cd6-4b50-4886-9616-750da4c045c9-kube-api-access-rpzcp\") pod \"6f0e8cd6-4b50-4886-9616-750da4c045c9\" (UID: \"6f0e8cd6-4b50-4886-9616-750da4c045c9\") "
Oct 30 23:22:26.884429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.884293    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-bundle\") pod \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\" (UID: \"b39c702b-05ee-40cb-8442-5cbc6ef62d1e\") "
Oct 30 23:22:26.885267 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.885211    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-bundle" pod "6f0e8cd6-4b50-4886-9616-750da4c045c9" (UID: "6f0e8cd6-4b50-4886-9616-750da4c045c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:26.889019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.888990    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-bundle" pod "b39c702b-05ee-40cb-8442-5cbc6ef62d1e" (UID: "b39c702b-05ee-40cb-8442-5cbc6ef62d1e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:26.891684 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.891655    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-kube-api-access-qf92p" pod "b39c702b-05ee-40cb-8442-5cbc6ef62d1e" (UID: "b39c702b-05ee-40cb-8442-5cbc6ef62d1e"). InnerVolumeSpecName "kube-api-access-qf92p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:26.892047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.892023    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-util" pod "b39c702b-05ee-40cb-8442-5cbc6ef62d1e" (UID: "b39c702b-05ee-40cb-8442-5cbc6ef62d1e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:26.892672 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.892630    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-util" pod "6f0e8cd6-4b50-4886-9616-750da4c045c9" (UID: "6f0e8cd6-4b50-4886-9616-750da4c045c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:26.893651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.893590    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0e8cd6-4b50-4886-9616-750da4c045c9-kube-api-access-rpzcp" pod "6f0e8cd6-4b50-4886-9616-750da4c045c9" (UID: "6f0e8cd6-4b50-4886-9616-750da4c045c9"). InnerVolumeSpecName "kube-api-access-rpzcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:26.985091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.984927    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpzcp\" (UniqueName: \"kubernetes.io/projected/6f0e8cd6-4b50-4886-9616-750da4c045c9-kube-api-access-rpzcp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:26.985091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.984976    4555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:26.985091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.984992    4555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:26.985091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.985006    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf92p\" (UniqueName: \"kubernetes.io/projected/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-kube-api-access-qf92p\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:26.985091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.985021    4555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f0e8cd6-4b50-4886-9616-750da4c045c9-util\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:26.985091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:26.985043    4555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b39c702b-05ee-40cb-8442-5cbc6ef62d1e-util\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:27.453074 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:22:27.453022    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:22:27.494888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:27.494823    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6"
Oct 30 23:22:27.495305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:27.494916    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qmzx6" event={"ID":"6f0e8cd6-4b50-4886-9616-750da4c045c9","Type":"ContainerDied","Data":"1dad373ca0f438df6a2e9edef9b1a5164c42871337b225d9a96962925995d43a"}
Oct 30 23:22:27.495305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:27.494956    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dad373ca0f438df6a2e9edef9b1a5164c42871337b225d9a96962925995d43a"
Oct 30 23:22:27.497989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:27.497961    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg"
Oct 30 23:22:27.497989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:27.497969    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mlllg" event={"ID":"b39c702b-05ee-40cb-8442-5cbc6ef62d1e","Type":"ContainerDied","Data":"e7113752e442971a047fa984ff4b856eca8593c1a9f3da865c7c65453a5bcfe5"}
Oct 30 23:22:27.498199 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:27.498008    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7113752e442971a047fa984ff4b856eca8593c1a9f3da865c7c65453a5bcfe5"
Oct 30 23:22:28.506953 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:28.506918    4555 generic.go:356] "Generic (PLEG): container finished" podID="e73d1c0e-a5c4-4166-828a-49c191300b22" containerID="896160c8ffa0b57b0d02fcc0a7e4f3c7c1d76d411858b89715ab4931f2444029" exitCode=0
Oct 30 23:22:28.507431 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:28.506997    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2" event={"ID":"e73d1c0e-a5c4-4166-828a-49c191300b22","Type":"ContainerDied","Data":"896160c8ffa0b57b0d02fcc0a7e4f3c7c1d76d411858b89715ab4931f2444029"}
Oct 30 23:22:28.509108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:28.509088    4555 generic.go:356] "Generic (PLEG): container finished" podID="118a6242-b02d-49fd-9d92-59e412aa1a5f" containerID="16a38924c8045c100bd7c8d9d70399b68e6e7c83bdbe767471139984ec1bb331" exitCode=0
Oct 30 23:22:28.509174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:28.509158    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6" event={"ID":"118a6242-b02d-49fd-9d92-59e412aa1a5f","Type":"ContainerDied","Data":"16a38924c8045c100bd7c8d9d70399b68e6e7c83bdbe767471139984ec1bb331"}
Oct 30 23:22:28.511075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:28.511048    4555 generic.go:356] "Generic (PLEG): container finished" podID="ab8e0839-7dfb-4255-9138-a041a06370cc" containerID="48ecd2f9966628bf653e158589bb0cc9f5b6a4f38b780864cd2e6992edc24daa" exitCode=0
Oct 30 23:22:28.511169 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:28.511135    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df" event={"ID":"ab8e0839-7dfb-4255-9138-a041a06370cc","Type":"ContainerDied","Data":"48ecd2f9966628bf653e158589bb0cc9f5b6a4f38b780864cd2e6992edc24daa"}
Oct 30 23:22:28.513264 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:28.513247    4555 generic.go:356] "Generic (PLEG): container finished" podID="17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" containerID="f37bf92de38326a234635f6018636bfad530abe3d84a781b9f9c9f9cf261d5b8" exitCode=0
Oct 30 23:22:28.513354 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:28.513331    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr" event={"ID":"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e","Type":"ContainerDied","Data":"f37bf92de38326a234635f6018636bfad530abe3d84a781b9f9c9f9cf261d5b8"}
Oct 30 23:22:28.515187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:28.515172    4555 generic.go:356] "Generic (PLEG): container finished" podID="1f7362fd-b371-4e8b-a428-9e548ebae04c" containerID="a497c958f0bfda3ae5df63a63b2ded7aae99acf0ac29e5e577b73b2b6b66e9bd" exitCode=0
Oct 30 23:22:28.515270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:28.515209    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm" event={"ID":"1f7362fd-b371-4e8b-a428-9e548ebae04c","Type":"ContainerDied","Data":"a497c958f0bfda3ae5df63a63b2ded7aae99acf0ac29e5e577b73b2b6b66e9bd"}
Oct 30 23:22:29.677209 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.677174    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:22:29.722711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.722636    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:22:29.726468 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.726445    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:22:29.731002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.730975    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:22:29.760125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.760099    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:22:29.814558 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.814525    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-kube-api-access-ffzg9\" (UniqueName: \"kubernetes.io/projected/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-kube-api-access-ffzg9\") pod \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") "
Oct 30 23:22:29.814746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.814568    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118a6242-b02d-49fd-9d92-59e412aa1a5f-kube-api-access-wmn2l\" (UniqueName: \"kubernetes.io/projected/118a6242-b02d-49fd-9d92-59e412aa1a5f-kube-api-access-wmn2l\") pod \"118a6242-b02d-49fd-9d92-59e412aa1a5f\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") "
Oct 30 23:22:29.814746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.814589    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-bundle\") pod \"e73d1c0e-a5c4-4166-828a-49c191300b22\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") "
Oct 30 23:22:29.814746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.814609    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7362fd-b371-4e8b-a428-9e548ebae04c-kube-api-access-hn7hm\" (UniqueName: \"kubernetes.io/projected/1f7362fd-b371-4e8b-a428-9e548ebae04c-kube-api-access-hn7hm\") pod \"1f7362fd-b371-4e8b-a428-9e548ebae04c\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") "
Oct 30 23:22:29.814746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.814629    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8e0839-7dfb-4255-9138-a041a06370cc-kube-api-access-7fst8\" (UniqueName: \"kubernetes.io/projected/ab8e0839-7dfb-4255-9138-a041a06370cc-kube-api-access-7fst8\") pod \"ab8e0839-7dfb-4255-9138-a041a06370cc\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") "
Oct 30 23:22:29.814746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.814651    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-bundle\") pod \"118a6242-b02d-49fd-9d92-59e412aa1a5f\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") "
Oct 30 23:22:29.815032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.814920    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-util\" (UniqueName: \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-util\") pod \"e73d1c0e-a5c4-4166-828a-49c191300b22\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") "
Oct 30 23:22:29.815032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.814972    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-bundle\") pod \"1f7362fd-b371-4e8b-a428-9e548ebae04c\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") "
Oct 30 23:22:29.815032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.815025    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-util\" (UniqueName: \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-util\") pod \"118a6242-b02d-49fd-9d92-59e412aa1a5f\" (UID: \"118a6242-b02d-49fd-9d92-59e412aa1a5f\") "
Oct 30 23:22:29.815189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.815065    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-bundle\") pod \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") "
Oct 30 23:22:29.815189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.815121    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-bundle" pod "118a6242-b02d-49fd-9d92-59e412aa1a5f" (UID: "118a6242-b02d-49fd-9d92-59e412aa1a5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:29.815189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.815137    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-util\" (UniqueName: \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-util\") pod \"1f7362fd-b371-4e8b-a428-9e548ebae04c\" (UID: \"1f7362fd-b371-4e8b-a428-9e548ebae04c\") "
Oct 30 23:22:29.815346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.815185    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73d1c0e-a5c4-4166-828a-49c191300b22-kube-api-access-xncf6\" (UniqueName: \"kubernetes.io/projected/e73d1c0e-a5c4-4166-828a-49c191300b22-kube-api-access-xncf6\") pod \"e73d1c0e-a5c4-4166-828a-49c191300b22\" (UID: \"e73d1c0e-a5c4-4166-828a-49c191300b22\") "
Oct 30 23:22:29.815346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.815227    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-util\" (UniqueName: \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-util\") pod \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\" (UID: \"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e\") "
Oct 30 23:22:29.815346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.815261    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-util\" (UniqueName: \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-util\") pod \"ab8e0839-7dfb-4255-9138-a041a06370cc\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") "
Oct 30 23:22:29.815346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.815301    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-bundle\") pod \"ab8e0839-7dfb-4255-9138-a041a06370cc\" (UID: \"ab8e0839-7dfb-4255-9138-a041a06370cc\") "
Oct 30 23:22:29.815577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.815553    4555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.815927 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.815249    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-bundle" pod "e73d1c0e-a5c4-4166-828a-49c191300b22" (UID: "e73d1c0e-a5c4-4166-828a-49c191300b22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:29.818722 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.818586    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-bundle" pod "ab8e0839-7dfb-4255-9138-a041a06370cc" (UID: "ab8e0839-7dfb-4255-9138-a041a06370cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:29.818722 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.818666    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8e0839-7dfb-4255-9138-a041a06370cc-kube-api-access-7fst8" pod "ab8e0839-7dfb-4255-9138-a041a06370cc" (UID: "ab8e0839-7dfb-4255-9138-a041a06370cc"). InnerVolumeSpecName "kube-api-access-7fst8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:29.818722 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.818686    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-kube-api-access-ffzg9" pod "17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" (UID: "17194046-7ad7-4ebf-b2ec-c53b70e5ba8e"). InnerVolumeSpecName "kube-api-access-ffzg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:29.819423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.819221    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-bundle" pod "17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" (UID: "17194046-7ad7-4ebf-b2ec-c53b70e5ba8e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:29.821922 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.821897    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7362fd-b371-4e8b-a428-9e548ebae04c-kube-api-access-hn7hm" pod "1f7362fd-b371-4e8b-a428-9e548ebae04c" (UID: "1f7362fd-b371-4e8b-a428-9e548ebae04c"). InnerVolumeSpecName "kube-api-access-hn7hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:29.826126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.826089    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-util" pod "e73d1c0e-a5c4-4166-828a-49c191300b22" (UID: "e73d1c0e-a5c4-4166-828a-49c191300b22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:29.826241 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.826219    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-util" pod "1f7362fd-b371-4e8b-a428-9e548ebae04c" (UID: "1f7362fd-b371-4e8b-a428-9e548ebae04c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:29.826492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.826468    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-bundle" pod "1f7362fd-b371-4e8b-a428-9e548ebae04c" (UID: "1f7362fd-b371-4e8b-a428-9e548ebae04c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:29.827008 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.826988    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-util" pod "17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" (UID: "17194046-7ad7-4ebf-b2ec-c53b70e5ba8e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:29.827326 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.827298    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118a6242-b02d-49fd-9d92-59e412aa1a5f-kube-api-access-wmn2l" pod "118a6242-b02d-49fd-9d92-59e412aa1a5f" (UID: "118a6242-b02d-49fd-9d92-59e412aa1a5f"). InnerVolumeSpecName "kube-api-access-wmn2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:29.827951 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.827927    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-util" pod "118a6242-b02d-49fd-9d92-59e412aa1a5f" (UID: "118a6242-b02d-49fd-9d92-59e412aa1a5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:29.828631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.828610    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-util" pod "ab8e0839-7dfb-4255-9138-a041a06370cc" (UID: "ab8e0839-7dfb-4255-9138-a041a06370cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:29.828834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.828814    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73d1c0e-a5c4-4166-828a-49c191300b22-kube-api-access-xncf6" pod "e73d1c0e-a5c4-4166-828a-49c191300b22" (UID: "e73d1c0e-a5c4-4166-828a-49c191300b22"). InnerVolumeSpecName "kube-api-access-xncf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:29.916112 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916079    4555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-util\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916112 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916107    4555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916112 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916117    4555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/118a6242-b02d-49fd-9d92-59e412aa1a5f-util\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916126    4555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916134    4555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f7362fd-b371-4e8b-a428-9e548ebae04c-util\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916145    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xncf6\" (UniqueName: \"kubernetes.io/projected/e73d1c0e-a5c4-4166-828a-49c191300b22-kube-api-access-xncf6\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916153    4555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-util\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916161    4555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-util\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916170    4555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab8e0839-7dfb-4255-9138-a041a06370cc-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916177    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffzg9\" (UniqueName: \"kubernetes.io/projected/17194046-7ad7-4ebf-b2ec-c53b70e5ba8e-kube-api-access-ffzg9\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916185    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmn2l\" (UniqueName: \"kubernetes.io/projected/118a6242-b02d-49fd-9d92-59e412aa1a5f-kube-api-access-wmn2l\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916193    4555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e73d1c0e-a5c4-4166-828a-49c191300b22-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916201    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn7hm\" (UniqueName: \"kubernetes.io/projected/1f7362fd-b371-4e8b-a428-9e548ebae04c-kube-api-access-hn7hm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:29.916341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:29.916209    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fst8\" (UniqueName: \"kubernetes.io/projected/ab8e0839-7dfb-4255-9138-a041a06370cc-kube-api-access-7fst8\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:30.524942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.524910    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm"
Oct 30 23:22:30.524942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.524918    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74q4nm" event={"ID":"1f7362fd-b371-4e8b-a428-9e548ebae04c","Type":"ContainerDied","Data":"57ca664fd5391cb34ae16d5fd1038fd774e88253e6497e2a9662dbf77c530ee9"}
Oct 30 23:22:30.525197 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.524970    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57ca664fd5391cb34ae16d5fd1038fd774e88253e6497e2a9662dbf77c530ee9"
Oct 30 23:22:30.526849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.526824    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2" event={"ID":"e73d1c0e-a5c4-4166-828a-49c191300b22","Type":"ContainerDied","Data":"1790328bc1addf2fa988ed8f9f6f187b48fa86e4934738ec289f865cca25f342"}
Oct 30 23:22:30.526849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.526838    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp72t2"
Oct 30 23:22:30.527032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.526861    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1790328bc1addf2fa988ed8f9f6f187b48fa86e4934738ec289f865cca25f342"
Oct 30 23:22:30.528870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.528851    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6"
Oct 30 23:22:30.528870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.528863    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62pmf6" event={"ID":"118a6242-b02d-49fd-9d92-59e412aa1a5f","Type":"ContainerDied","Data":"9cddec2af070dd2ec319e019d3a9327276e6907d3bdd51fe86039c7c257a8398"}
Oct 30 23:22:30.529061 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.528896    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cddec2af070dd2ec319e019d3a9327276e6907d3bdd51fe86039c7c257a8398"
Oct 30 23:22:30.531294 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.531266    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df" event={"ID":"ab8e0839-7dfb-4255-9138-a041a06370cc","Type":"ContainerDied","Data":"2671acae46fa66ccfcbda478a05aed2d9a16b7bd57e23c6868bff3120da982f0"}
Oct 30 23:22:30.531423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.531310    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2671acae46fa66ccfcbda478a05aed2d9a16b7bd57e23c6868bff3120da982f0"
Oct 30 23:22:30.531423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.531363    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbf6df"
Oct 30 23:22:30.533531 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.533510    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr" event={"ID":"17194046-7ad7-4ebf-b2ec-c53b70e5ba8e","Type":"ContainerDied","Data":"a36b457af1c51c1c2ae2842d8e76a059b98934528e27174aa3774a73a4118246"}
Oct 30 23:22:30.533615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.533548    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a36b457af1c51c1c2ae2842d8e76a059b98934528e27174aa3774a73a4118246"
Oct 30 23:22:30.533615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:30.533579    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fkrr"
Oct 30 23:22:31.216348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216306    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"]
Oct 30 23:22:31.216753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216727    4555 cpu_manager.go:414] "containerMap: removing container" podUID="6f0e8cd6-4b50-4886-9616-750da4c045c9" containerName="pull"
Oct 30 23:22:31.216753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216744    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0e8cd6-4b50-4886-9616-750da4c045c9" containerName="pull"
Oct 30 23:22:31.216829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216762    4555 cpu_manager.go:414] "containerMap: removing container" podUID="e73d1c0e-a5c4-4166-828a-49c191300b22" containerName="extract"
Oct 30 23:22:31.216829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216773    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73d1c0e-a5c4-4166-828a-49c191300b22" containerName="extract"
Oct 30 23:22:31.216829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216790    4555 cpu_manager.go:414] "containerMap: removing container" podUID="1f7362fd-b371-4e8b-a428-9e548ebae04c" containerName="extract"
Oct 30 23:22:31.216829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216797    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7362fd-b371-4e8b-a428-9e548ebae04c" containerName="extract"
Oct 30 23:22:31.216829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216809    4555 cpu_manager.go:414] "containerMap: removing container" podUID="118a6242-b02d-49fd-9d92-59e412aa1a5f" containerName="util"
Oct 30 23:22:31.216829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216815    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118a6242-b02d-49fd-9d92-59e412aa1a5f" containerName="util"
Oct 30 23:22:31.216829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216826    4555 cpu_manager.go:414] "containerMap: removing container" podUID="40a92c96-b669-4712-8362-47f991b6555a" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216835    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a92c96-b669-4712-8362-47f991b6555a" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216849    4555 cpu_manager.go:414] "containerMap: removing container" podUID="40a92c96-b669-4712-8362-47f991b6555a" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216855    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a92c96-b669-4712-8362-47f991b6555a" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216866    4555 cpu_manager.go:414] "containerMap: removing container" podUID="e73d1c0e-a5c4-4166-828a-49c191300b22" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216873    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73d1c0e-a5c4-4166-828a-49c191300b22" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216885    4555 cpu_manager.go:414] "containerMap: removing container" podUID="6f0e8cd6-4b50-4886-9616-750da4c045c9" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216892    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0e8cd6-4b50-4886-9616-750da4c045c9" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216901    4555 cpu_manager.go:414] "containerMap: removing container" podUID="17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216908    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216921    4555 cpu_manager.go:414] "containerMap: removing container" podUID="17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216928    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216936    4555 cpu_manager.go:414] "containerMap: removing container" podUID="17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216942    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216954    4555 cpu_manager.go:414] "containerMap: removing container" podUID="1f7362fd-b371-4e8b-a428-9e548ebae04c" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216960    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7362fd-b371-4e8b-a428-9e548ebae04c" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216969    4555 cpu_manager.go:414] "containerMap: removing container" podUID="b39c702b-05ee-40cb-8442-5cbc6ef62d1e" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216976    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39c702b-05ee-40cb-8442-5cbc6ef62d1e" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216987    4555 cpu_manager.go:414] "containerMap: removing container" podUID="40a92c96-b669-4712-8362-47f991b6555a" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216992    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a92c96-b669-4712-8362-47f991b6555a" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.216998    4555 cpu_manager.go:414] "containerMap: removing container" podUID="1f7362fd-b371-4e8b-a428-9e548ebae04c" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217004    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7362fd-b371-4e8b-a428-9e548ebae04c" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217011    4555 cpu_manager.go:414] "containerMap: removing container" podUID="6f0e8cd6-4b50-4886-9616-750da4c045c9" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217015    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0e8cd6-4b50-4886-9616-750da4c045c9" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217025    4555 cpu_manager.go:414] "containerMap: removing container" podUID="e73d1c0e-a5c4-4166-828a-49c191300b22" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217037    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73d1c0e-a5c4-4166-828a-49c191300b22" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217044    4555 cpu_manager.go:414] "containerMap: removing container" podUID="ab8e0839-7dfb-4255-9138-a041a06370cc" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217049    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8e0839-7dfb-4255-9138-a041a06370cc" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217056    4555 cpu_manager.go:414] "containerMap: removing container" podUID="ab8e0839-7dfb-4255-9138-a041a06370cc" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217061    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8e0839-7dfb-4255-9138-a041a06370cc" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217066    4555 cpu_manager.go:414] "containerMap: removing container" podUID="118a6242-b02d-49fd-9d92-59e412aa1a5f" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217069    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118a6242-b02d-49fd-9d92-59e412aa1a5f" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217075    4555 cpu_manager.go:414] "containerMap: removing container" podUID="ab8e0839-7dfb-4255-9138-a041a06370cc" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217079    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8e0839-7dfb-4255-9138-a041a06370cc" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217085    4555 cpu_manager.go:414] "containerMap: removing container" podUID="118a6242-b02d-49fd-9d92-59e412aa1a5f" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217089    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118a6242-b02d-49fd-9d92-59e412aa1a5f" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217095    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7faba7c5-348f-4137-8e79-f220d806181c" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217099    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7faba7c5-348f-4137-8e79-f220d806181c" containerName="util"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217105    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7faba7c5-348f-4137-8e79-f220d806181c" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217109    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7faba7c5-348f-4137-8e79-f220d806181c" containerName="pull"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217115    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7faba7c5-348f-4137-8e79-f220d806181c" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217120    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7faba7c5-348f-4137-8e79-f220d806181c" containerName="extract"
Oct 30 23:22:31.217107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217127    4555 cpu_manager.go:414] "containerMap: removing container" podUID="b39c702b-05ee-40cb-8442-5cbc6ef62d1e" containerName="pull"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217133    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39c702b-05ee-40cb-8442-5cbc6ef62d1e" containerName="pull"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217143    4555 cpu_manager.go:414] "containerMap: removing container" podUID="b39c702b-05ee-40cb-8442-5cbc6ef62d1e" containerName="util"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217147    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39c702b-05ee-40cb-8442-5cbc6ef62d1e" containerName="util"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217213    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39c702b-05ee-40cb-8442-5cbc6ef62d1e" containerName="extract"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217226    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0e8cd6-4b50-4886-9616-750da4c045c9" containerName="extract"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217235    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="118a6242-b02d-49fd-9d92-59e412aa1a5f" containerName="extract"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217242    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8e0839-7dfb-4255-9138-a041a06370cc" containerName="extract"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217248    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="7faba7c5-348f-4137-8e79-f220d806181c" containerName="extract"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217253    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73d1c0e-a5c4-4166-828a-49c191300b22" containerName="extract"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217259    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7362fd-b371-4e8b-a428-9e548ebae04c" containerName="extract"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217264    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a92c96-b669-4712-8362-47f991b6555a" containerName="extract"
Oct 30 23:22:31.218448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.217270    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="17194046-7ad7-4ebf-b2ec-c53b70e5ba8e" containerName="extract"
Oct 30 23:22:31.221516 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.221499    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.235679 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.235649    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 23:22:31.235679 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.235659    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 23:22:31.235897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.235709    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 23:22:31.235897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.235793    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2mrzn\""
Oct 30 23:22:31.235897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.235802    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 23:22:31.263061 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.263031    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"]
Oct 30 23:22:31.327917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.327880    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/feb8e67e-7507-428f-a274-8793dee4b0d3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.328086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.327931    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/feb8e67e-7507-428f-a274-8793dee4b0d3-metrics-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.328086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.328013    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/feb8e67e-7507-428f-a274-8793dee4b0d3-manifests\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.328086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.328046    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feb8e67e-7507-428f-a274-8793dee4b0d3-tmp\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.328285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.328117    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/feb8e67e-7507-428f-a274-8793dee4b0d3-webhook-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.328285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.328208    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mrj\" (UniqueName: \"kubernetes.io/projected/feb8e67e-7507-428f-a274-8793dee4b0d3-kube-api-access-69mrj\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.341440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.341419    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 23:22:31.428997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.428964    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/feb8e67e-7507-428f-a274-8793dee4b0d3-webhook-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.429167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.429003    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mrj\" (UniqueName: \"kubernetes.io/projected/feb8e67e-7507-428f-a274-8793dee4b0d3-kube-api-access-69mrj\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.429167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.429059    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/feb8e67e-7507-428f-a274-8793dee4b0d3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.429167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.429107    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/feb8e67e-7507-428f-a274-8793dee4b0d3-metrics-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.429332 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.429242    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/feb8e67e-7507-428f-a274-8793dee4b0d3-manifests\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.429332 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.429315    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feb8e67e-7507-428f-a274-8793dee4b0d3-tmp\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.429673 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.429647    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/feb8e67e-7507-428f-a274-8793dee4b0d3-manifests\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.431689 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.431661    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feb8e67e-7507-428f-a274-8793dee4b0d3-tmp\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.431795 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.431701    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/feb8e67e-7507-428f-a274-8793dee4b0d3-webhook-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.431870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.431854    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/feb8e67e-7507-428f-a274-8793dee4b0d3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.431908 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.431860    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/feb8e67e-7507-428f-a274-8793dee4b0d3-metrics-cert\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.442700 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.442676    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mrj\" (UniqueName: \"kubernetes.io/projected/feb8e67e-7507-428f-a274-8793dee4b0d3-kube-api-access-69mrj\") pod \"opendatahub-operator-controller-manager-6986c789f7-kp2lg\" (UID: \"feb8e67e-7507-428f-a274-8793dee4b0d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.536696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.536666    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:31.842628 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:31.842592    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"]
Oct 30 23:22:31.843782 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:22:31.843755    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb8e67e_7507_428f_a274_8793dee4b0d3.slice/crio-e03f809087df849012c47af6c6d65b2668810c95e51df3ab7c054a0766e64490 WatchSource:0}: Error finding container e03f809087df849012c47af6c6d65b2668810c95e51df3ab7c054a0766e64490: Status 404 returned error can't find the container with id e03f809087df849012c47af6c6d65b2668810c95e51df3ab7c054a0766e64490
Oct 30 23:22:32.545048 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:32.545004    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg" event={"ID":"feb8e67e-7507-428f-a274-8793dee4b0d3","Type":"ContainerStarted","Data":"e03f809087df849012c47af6c6d65b2668810c95e51df3ab7c054a0766e64490"}
Oct 30 23:22:35.561945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:35.561906    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg" event={"ID":"feb8e67e-7507-428f-a274-8793dee4b0d3","Type":"ContainerStarted","Data":"a8c6a8864b4fb35e02dbf62265c0cebd5d2f02296139ccde59a14406eb2c8a9f"}
Oct 30 23:22:36.568057 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:36.568020    4555 generic.go:356] "Generic (PLEG): container finished" podID="feb8e67e-7507-428f-a274-8793dee4b0d3" containerID="a8c6a8864b4fb35e02dbf62265c0cebd5d2f02296139ccde59a14406eb2c8a9f" exitCode=0
Oct 30 23:22:36.568493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:36.568064    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg" event={"ID":"feb8e67e-7507-428f-a274-8793dee4b0d3","Type":"ContainerDied","Data":"a8c6a8864b4fb35e02dbf62265c0cebd5d2f02296139ccde59a14406eb2c8a9f"}
Oct 30 23:22:37.574309 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:37.574269    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg" event={"ID":"feb8e67e-7507-428f-a274-8793dee4b0d3","Type":"ContainerStarted","Data":"71184ff26698f1e77f38e73f9967accc5f97fd4a32bd4c9f11a8f07b5b41782f"}
Oct 30 23:22:37.574819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:37.574412    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:37.682377 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:37.682328    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg" podStartSLOduration=2.239672343 podStartE2EDuration="6.682310716s" totalImagesPullingTime="3.998314754s" totalInitContainerRuntime="444.323619ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:37.679705689 +0000 UTC m=+216.606343273" watchObservedRunningTime="2026-10-30 23:22:37.682310716 +0000 UTC m=+216.608948317"
Oct 30 23:22:42.392633 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.392554    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c"]
Oct 30 23:22:42.395688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.395672    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c"
Oct 30 23:22:42.399587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.399564    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 23:22:42.399587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.399582    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 23:22:42.400626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.400610    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8hl8w\""
Oct 30 23:22:42.406122 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.406100    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c"]
Oct 30 23:22:42.529742 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.529702    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4fr4\" (UniqueName: \"kubernetes.io/projected/67c3b2ff-66e7-466f-a332-8e6b96ec54cf-kube-api-access-m4fr4\") pod \"authorino-operator-7b7bb4bc77-fjn4c\" (UID: \"67c3b2ff-66e7-466f-a332-8e6b96ec54cf\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c"
Oct 30 23:22:42.630749 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.630719    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4fr4\" (UniqueName: \"kubernetes.io/projected/67c3b2ff-66e7-466f-a332-8e6b96ec54cf-kube-api-access-m4fr4\") pod \"authorino-operator-7b7bb4bc77-fjn4c\" (UID: \"67c3b2ff-66e7-466f-a332-8e6b96ec54cf\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c"
Oct 30 23:22:42.657036 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.656966    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4fr4\" (UniqueName: \"kubernetes.io/projected/67c3b2ff-66e7-466f-a332-8e6b96ec54cf-kube-api-access-m4fr4\") pod \"authorino-operator-7b7bb4bc77-fjn4c\" (UID: \"67c3b2ff-66e7-466f-a332-8e6b96ec54cf\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c"
Oct 30 23:22:42.708529 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.708489    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c"
Oct 30 23:22:42.990382 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:42.990343    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c"]
Oct 30 23:22:43.600402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:43.600353    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c" event={"ID":"67c3b2ff-66e7-466f-a332-8e6b96ec54cf","Type":"ContainerStarted","Data":"45cae73e8005b8bbe2dee429a0bcbce9cd9af2d35451e946c156bc043da93745"}
Oct 30 23:22:46.616684 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:46.616652    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c" event={"ID":"67c3b2ff-66e7-466f-a332-8e6b96ec54cf","Type":"ContainerStarted","Data":"41af10178584c9f24deed7728d322f14ebdf4431a472a0ea5536180e2166c659"}
Oct 30 23:22:46.617099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:46.616772    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c"
Oct 30 23:22:46.659939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:46.659885    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c" podStartSLOduration=1.8667500559999999 podStartE2EDuration="4.659857186s" totalImagesPullingTime="2.79310713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:46.658966181 +0000 UTC m=+225.585603762" watchObservedRunningTime="2026-10-30 23:22:46.659857186 +0000 UTC m=+225.586494768"
Oct 30 23:22:47.049913 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.049874    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn"]
Oct 30 23:22:47.053310 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.053289    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn"
Oct 30 23:22:47.067855 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.067836    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rfmrw\""
Oct 30 23:22:47.068598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.068586    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 23:22:47.079509 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.079488    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn"]
Oct 30 23:22:47.169334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.169297    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v2kj\" (UniqueName: \"kubernetes.io/projected/e28503fb-c336-451f-a3a0-bd3e8d0673a3-kube-api-access-7v2kj\") pod \"dns-operator-controller-manager-8557cbdb84-5fdkn\" (UID: \"e28503fb-c336-451f-a3a0-bd3e8d0673a3\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn"
Oct 30 23:22:47.269866 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.269827    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v2kj\" (UniqueName: \"kubernetes.io/projected/e28503fb-c336-451f-a3a0-bd3e8d0673a3-kube-api-access-7v2kj\") pod \"dns-operator-controller-manager-8557cbdb84-5fdkn\" (UID: \"e28503fb-c336-451f-a3a0-bd3e8d0673a3\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn"
Oct 30 23:22:47.289069 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.289041    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v2kj\" (UniqueName: \"kubernetes.io/projected/e28503fb-c336-451f-a3a0-bd3e8d0673a3-kube-api-access-7v2kj\") pod \"dns-operator-controller-manager-8557cbdb84-5fdkn\" (UID: \"e28503fb-c336-451f-a3a0-bd3e8d0673a3\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn"
Oct 30 23:22:47.365806 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.365708    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn"
Oct 30 23:22:47.645636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.645610    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn"]
Oct 30 23:22:47.648139 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:22:47.648113    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28503fb_c336_451f_a3a0_bd3e8d0673a3.slice/crio-e35e4d4e8e2bbed36ab14d7ee31f329893e574fb2110e602b6eb311fbff9d0ef WatchSource:0}: Error finding container e35e4d4e8e2bbed36ab14d7ee31f329893e574fb2110e602b6eb311fbff9d0ef: Status 404 returned error can't find the container with id e35e4d4e8e2bbed36ab14d7ee31f329893e574fb2110e602b6eb311fbff9d0ef
Oct 30 23:22:47.793216 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.793183    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"]
Oct 30 23:22:47.797713 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.797697    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"
Oct 30 23:22:47.801741 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.801702    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:22:47.802000 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.801986    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 23:22:47.802287 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.802269    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qdwfl\""
Oct 30 23:22:47.816116 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.816088    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"]
Oct 30 23:22:47.977623 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.977475    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90a4f524-2f55-47f2-834b-6f437d2ca67e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6ph24\" (UID: \"90a4f524-2f55-47f2-834b-6f437d2ca67e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"
Oct 30 23:22:47.977623 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:47.977542    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xs7z\" (UniqueName: \"kubernetes.io/projected/90a4f524-2f55-47f2-834b-6f437d2ca67e-kube-api-access-7xs7z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6ph24\" (UID: \"90a4f524-2f55-47f2-834b-6f437d2ca67e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"
Oct 30 23:22:48.078686 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:48.078651    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90a4f524-2f55-47f2-834b-6f437d2ca67e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6ph24\" (UID: \"90a4f524-2f55-47f2-834b-6f437d2ca67e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"
Oct 30 23:22:48.078867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:48.078700    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xs7z\" (UniqueName: \"kubernetes.io/projected/90a4f524-2f55-47f2-834b-6f437d2ca67e-kube-api-access-7xs7z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6ph24\" (UID: \"90a4f524-2f55-47f2-834b-6f437d2ca67e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"
Oct 30 23:22:48.079114 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:48.079093    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90a4f524-2f55-47f2-834b-6f437d2ca67e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6ph24\" (UID: \"90a4f524-2f55-47f2-834b-6f437d2ca67e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"
Oct 30 23:22:48.091631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:48.091607    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xs7z\" (UniqueName: \"kubernetes.io/projected/90a4f524-2f55-47f2-834b-6f437d2ca67e-kube-api-access-7xs7z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6ph24\" (UID: \"90a4f524-2f55-47f2-834b-6f437d2ca67e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"
Oct 30 23:22:48.114585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:48.114553    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"
Oct 30 23:22:48.405499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:48.405472    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24"]
Oct 30 23:22:48.407829 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:22:48.407789    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a4f524_2f55_47f2_834b_6f437d2ca67e.slice/crio-e6c588872adca0e8f4b215d8cb661e20bb0793ba445948b53b38cbed2826e180 WatchSource:0}: Error finding container e6c588872adca0e8f4b215d8cb661e20bb0793ba445948b53b38cbed2826e180: Status 404 returned error can't find the container with id e6c588872adca0e8f4b215d8cb661e20bb0793ba445948b53b38cbed2826e180
Oct 30 23:22:48.581797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:48.581761    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6986c789f7-kp2lg"
Oct 30 23:22:48.628577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:48.628533    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24" event={"ID":"90a4f524-2f55-47f2-834b-6f437d2ca67e","Type":"ContainerStarted","Data":"e6c588872adca0e8f4b215d8cb661e20bb0793ba445948b53b38cbed2826e180"}
Oct 30 23:22:48.630039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:48.630014    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn" event={"ID":"e28503fb-c336-451f-a3a0-bd3e8d0673a3","Type":"ContainerStarted","Data":"e35e4d4e8e2bbed36ab14d7ee31f329893e574fb2110e602b6eb311fbff9d0ef"}
Oct 30 23:22:49.637554 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:49.637514    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn" event={"ID":"e28503fb-c336-451f-a3a0-bd3e8d0673a3","Type":"ContainerStarted","Data":"8c53e1f745cf882bd8eeb4b6cf09e6f0848a1dbe3d8e7335419d0c181d7e3b00"}
Oct 30 23:22:49.637980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:49.637668    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn"
Oct 30 23:22:49.670555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:49.670459    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn" podStartSLOduration=1.92110701 podStartE2EDuration="3.670443722s" totalImagesPullingTime="1.749336712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:49.668950634 +0000 UTC m=+228.595588215" watchObservedRunningTime="2026-10-30 23:22:49.670443722 +0000 UTC m=+228.597081303"
Oct 30 23:22:50.644422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:50.644365    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24" event={"ID":"90a4f524-2f55-47f2-834b-6f437d2ca67e","Type":"ContainerStarted","Data":"de0a1ebbbbfac44406f493b826323bdb7ade281b1141d220a611429bf76e6b96"}
Oct 30 23:22:50.689210 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:50.689156    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6ph24" podStartSLOduration=1.591184484 podStartE2EDuration="3.689141034s" totalImagesPullingTime="2.09795655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:50.688196105 +0000 UTC m=+229.614833687" watchObservedRunningTime="2026-10-30 23:22:50.689141034 +0000 UTC m=+229.615778615"
Oct 30 23:22:54.597652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.597610    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"]
Oct 30 23:22:54.601267 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.601248    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:22:54.606115 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.606090    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4vmw2\""
Oct 30 23:22:54.621408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.621349    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"]
Oct 30 23:22:54.638028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.637998    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpp2q\" (UniqueName: \"kubernetes.io/projected/02b04936-0271-4d0d-b14e-fc53c9b85bff-kube-api-access-rpp2q\") pod \"kuadrant-operator-controller-manager-678d87766f-q7bv9\" (UID: \"02b04936-0271-4d0d-b14e-fc53c9b85bff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:22:54.638193 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.638041    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02b04936-0271-4d0d-b14e-fc53c9b85bff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q7bv9\" (UID: \"02b04936-0271-4d0d-b14e-fc53c9b85bff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:22:54.739066 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.739010    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpp2q\" (UniqueName: \"kubernetes.io/projected/02b04936-0271-4d0d-b14e-fc53c9b85bff-kube-api-access-rpp2q\") pod \"kuadrant-operator-controller-manager-678d87766f-q7bv9\" (UID: \"02b04936-0271-4d0d-b14e-fc53c9b85bff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:22:54.739066 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.739073    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02b04936-0271-4d0d-b14e-fc53c9b85bff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q7bv9\" (UID: \"02b04936-0271-4d0d-b14e-fc53c9b85bff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:22:54.739478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.739462    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02b04936-0271-4d0d-b14e-fc53c9b85bff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q7bv9\" (UID: \"02b04936-0271-4d0d-b14e-fc53c9b85bff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:22:54.752564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.752529    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpp2q\" (UniqueName: \"kubernetes.io/projected/02b04936-0271-4d0d-b14e-fc53c9b85bff-kube-api-access-rpp2q\") pod \"kuadrant-operator-controller-manager-678d87766f-q7bv9\" (UID: \"02b04936-0271-4d0d-b14e-fc53c9b85bff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:22:54.915469 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:54.915353    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:22:55.206584 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:22:55.206556    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b04936_0271_4d0d_b14e_fc53c9b85bff.slice/crio-c1fffa5b142b441c3c9fb00f67885a604481ae7413c2bd2dbe7671601d990d79 WatchSource:0}: Error finding container c1fffa5b142b441c3c9fb00f67885a604481ae7413c2bd2dbe7671601d990d79: Status 404 returned error can't find the container with id c1fffa5b142b441c3c9fb00f67885a604481ae7413c2bd2dbe7671601d990d79
Oct 30 23:22:55.206788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:55.206769    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"]
Oct 30 23:22:55.668490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:55.668457    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9" event={"ID":"02b04936-0271-4d0d-b14e-fc53c9b85bff","Type":"ContainerStarted","Data":"c1fffa5b142b441c3c9fb00f67885a604481ae7413c2bd2dbe7671601d990d79"}
Oct 30 23:22:57.450086 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:22:57.450037    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:22:57.623693 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:57.623666    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fjn4c"
Oct 30 23:22:58.140883 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:58.140831    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r"]
Oct 30 23:22:58.145009 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:58.144985    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r"
Oct 30 23:22:58.155513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:58.155486    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-r29vd\""
Oct 30 23:22:58.167057 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:58.167024    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r"]
Oct 30 23:22:58.169638 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:58.169605    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvmb\" (UniqueName: \"kubernetes.io/projected/bb713bda-562e-4214-976d-1bac69ba3efd-kube-api-access-wcvmb\") pod \"limitador-operator-controller-manager-5778bb4449-x4x9r\" (UID: \"bb713bda-562e-4214-976d-1bac69ba3efd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r"
Oct 30 23:22:58.270862 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:58.270813    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvmb\" (UniqueName: \"kubernetes.io/projected/bb713bda-562e-4214-976d-1bac69ba3efd-kube-api-access-wcvmb\") pod \"limitador-operator-controller-manager-5778bb4449-x4x9r\" (UID: \"bb713bda-562e-4214-976d-1bac69ba3efd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r"
Oct 30 23:22:58.286722 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:58.286688    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvmb\" (UniqueName: \"kubernetes.io/projected/bb713bda-562e-4214-976d-1bac69ba3efd-kube-api-access-wcvmb\") pod \"limitador-operator-controller-manager-5778bb4449-x4x9r\" (UID: \"bb713bda-562e-4214-976d-1bac69ba3efd\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r"
Oct 30 23:22:58.458396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:58.458290    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r"
Oct 30 23:22:58.745990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:58.745964    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r"]
Oct 30 23:22:58.748634 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:22:58.748592    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb713bda_562e_4214_976d_1bac69ba3efd.slice/crio-39c893be059494fefb55da109f7cf0734edfcb4e694fddf910a554dfadd40549 WatchSource:0}: Error finding container 39c893be059494fefb55da109f7cf0734edfcb4e694fddf910a554dfadd40549: Status 404 returned error can't find the container with id 39c893be059494fefb55da109f7cf0734edfcb4e694fddf910a554dfadd40549
Oct 30 23:22:59.690193 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:59.690149    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r" event={"ID":"bb713bda-562e-4214-976d-1bac69ba3efd","Type":"ContainerStarted","Data":"39c893be059494fefb55da109f7cf0734edfcb4e694fddf910a554dfadd40549"}
Oct 30 23:22:59.692404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:59.692349    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9" event={"ID":"02b04936-0271-4d0d-b14e-fc53c9b85bff","Type":"ContainerStarted","Data":"4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10"}
Oct 30 23:22:59.692565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:59.692494    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:22:59.728124 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:22:59.728075    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9" podStartSLOduration=1.434387331 podStartE2EDuration="5.728059529s" totalImagesPullingTime="4.293672198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:59.725604373 +0000 UTC m=+238.652241948" watchObservedRunningTime="2026-10-30 23:22:59.728059529 +0000 UTC m=+238.654697110"
Oct 30 23:23:00.629155 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.629125    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"]
Oct 30 23:23:00.633429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.633190    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.646765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.646739    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5fdkn"
Oct 30 23:23:00.657544 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.657511    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8l2q9\""
Oct 30 23:23:00.659699 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.659670    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"]
Oct 30 23:23:00.694072 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.694042    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.694585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.694092    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.694585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.694129    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dcdt\" (UniqueName: \"kubernetes.io/projected/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-kube-api-access-2dcdt\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.694585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.694164    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.694585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.694211    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.694585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.694244    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.694585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.694280    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.694585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.694360    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.694585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.694422    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.794975 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.794950    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.795115 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.795022    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.795115 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.795080    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dcdt\" (UniqueName: \"kubernetes.io/projected/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-kube-api-access-2dcdt\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.795209 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.795162    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.795255 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.795221    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.795308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.795256    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.795308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.795292    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.795426 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.795359    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.795426 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.795408    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.795812 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.795785    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.795922 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.795865    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.796148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.796126    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.796243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.796222    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.796243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.796237    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.798186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.798161    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.798342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.798323    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.812542 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.812515    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.812858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.812836    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dcdt\" (UniqueName: \"kubernetes.io/projected/91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2-kube-api-access-2dcdt\") pod \"data-science-gateway-data-science-gateway-class-56c7694f859hrmz\" (UID: \"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:00.949623 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:00.949534    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:01.257160 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:23:01.257125    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91db73c5_86c5_4ec1_a7a8_dbdd5effa7b2.slice/crio-584b41d32e5ac2d3842d46a284c65c88489be41b5f99bc390542bed0159f5d9f WatchSource:0}: Error finding container 584b41d32e5ac2d3842d46a284c65c88489be41b5f99bc390542bed0159f5d9f: Status 404 returned error can't find the container with id 584b41d32e5ac2d3842d46a284c65c88489be41b5f99bc390542bed0159f5d9f
Oct 30 23:23:01.259415 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.259381    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:23:01.259502 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.259450    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:23:01.259502 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.259478    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:23:01.267658 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.267637    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"]
Oct 30 23:23:01.703632 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.703597    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r" event={"ID":"bb713bda-562e-4214-976d-1bac69ba3efd","Type":"ContainerStarted","Data":"0b623f15f1c0d691baa45e78bc956068ea9cc38051d91540e44833865eb1c9ac"}
Oct 30 23:23:01.704117 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.703667    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r"
Oct 30 23:23:01.705248 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.705226    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz" event={"ID":"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2","Type":"ContainerStarted","Data":"43b41235eff097e2be08a94c4eb3f36d04cbcf4417600b9b5d91581d1503dd39"}
Oct 30 23:23:01.705394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.705261    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz" event={"ID":"91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2","Type":"ContainerStarted","Data":"584b41d32e5ac2d3842d46a284c65c88489be41b5f99bc390542bed0159f5d9f"}
Oct 30 23:23:01.728164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.728110    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r" podStartSLOduration=1.7308993419999998 podStartE2EDuration="3.72809614s" totalImagesPullingTime="1.997196798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:01.726009203 +0000 UTC m=+240.652646785" watchObservedRunningTime="2026-10-30 23:23:01.72809614 +0000 UTC m=+240.654733721"
Oct 30 23:23:01.761813 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.761748    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz" podStartSLOduration=1.7617280370000001 podStartE2EDuration="1.761728037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:01.75991457 +0000 UTC m=+240.686552151" watchObservedRunningTime="2026-10-30 23:23:01.761728037 +0000 UTC m=+240.688365620"
Oct 30 23:23:01.950192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:01.950160    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:02.955050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:02.955023    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:03.715518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:03.715488    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:03.716558 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:03.716539    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f859hrmz"
Oct 30 23:23:10.699719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:10.699689    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:23:12.713340 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:12.713310    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x4x9r"
Oct 30 23:23:27.445135 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:23:27.445098    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:23:56.818782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.818746    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pm8s9"]
Oct 30 23:23:56.822324 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.822301    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9"
Oct 30 23:23:56.832796 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.832771    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pm8s9"]
Oct 30 23:23:56.833907 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.832854    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 23:23:56.833907 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.833237    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 23:23:56.838083 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.838056    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 23:23:56.838208 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.838124    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-snp6s\""
Oct 30 23:23:56.839015 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.838995    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"]
Oct 30 23:23:56.842729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.842711    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:23:56.853753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.853726    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"]
Oct 30 23:23:56.881134 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.881103    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06caa6ed-2a57-49ea-becf-ac4d8088e260-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-46r5z\" (UID: \"06caa6ed-2a57-49ea-becf-ac4d8088e260\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:23:56.881311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.881155    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b47baf-3103-4d30-953e-62628d4a795c-cert\") pod \"kserve-controller-manager-776cc66d47-pm8s9\" (UID: \"78b47baf-3103-4d30-953e-62628d4a795c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9"
Oct 30 23:23:56.881311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.881239    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45nr\" (UniqueName: \"kubernetes.io/projected/06caa6ed-2a57-49ea-becf-ac4d8088e260-kube-api-access-j45nr\") pod \"llmisvc-controller-manager-869d5dcc5d-46r5z\" (UID: \"06caa6ed-2a57-49ea-becf-ac4d8088e260\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:23:56.881311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.881282    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmr56\" (UniqueName: \"kubernetes.io/projected/78b47baf-3103-4d30-953e-62628d4a795c-kube-api-access-vmr56\") pod \"kserve-controller-manager-776cc66d47-pm8s9\" (UID: \"78b47baf-3103-4d30-953e-62628d4a795c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9"
Oct 30 23:23:56.930500 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.930466    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 23:23:56.930742 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.930722    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cqm5s\""
Oct 30 23:23:56.981698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.981664    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45nr\" (UniqueName: \"kubernetes.io/projected/06caa6ed-2a57-49ea-becf-ac4d8088e260-kube-api-access-j45nr\") pod \"llmisvc-controller-manager-869d5dcc5d-46r5z\" (UID: \"06caa6ed-2a57-49ea-becf-ac4d8088e260\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:23:56.981887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.981705    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmr56\" (UniqueName: \"kubernetes.io/projected/78b47baf-3103-4d30-953e-62628d4a795c-kube-api-access-vmr56\") pod \"kserve-controller-manager-776cc66d47-pm8s9\" (UID: \"78b47baf-3103-4d30-953e-62628d4a795c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9"
Oct 30 23:23:56.981887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.981746    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06caa6ed-2a57-49ea-becf-ac4d8088e260-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-46r5z\" (UID: \"06caa6ed-2a57-49ea-becf-ac4d8088e260\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:23:56.981887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.981778    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b47baf-3103-4d30-953e-62628d4a795c-cert\") pod \"kserve-controller-manager-776cc66d47-pm8s9\" (UID: \"78b47baf-3103-4d30-953e-62628d4a795c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9"
Oct 30 23:23:56.984356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.984334    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06caa6ed-2a57-49ea-becf-ac4d8088e260-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-46r5z\" (UID: \"06caa6ed-2a57-49ea-becf-ac4d8088e260\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:23:56.984511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.984412    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b47baf-3103-4d30-953e-62628d4a795c-cert\") pod \"kserve-controller-manager-776cc66d47-pm8s9\" (UID: \"78b47baf-3103-4d30-953e-62628d4a795c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9"
Oct 30 23:23:56.996760 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.996733    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45nr\" (UniqueName: \"kubernetes.io/projected/06caa6ed-2a57-49ea-becf-ac4d8088e260-kube-api-access-j45nr\") pod \"llmisvc-controller-manager-869d5dcc5d-46r5z\" (UID: \"06caa6ed-2a57-49ea-becf-ac4d8088e260\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:23:56.996878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:56.996828    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmr56\" (UniqueName: \"kubernetes.io/projected/78b47baf-3103-4d30-953e-62628d4a795c-kube-api-access-vmr56\") pod \"kserve-controller-manager-776cc66d47-pm8s9\" (UID: \"78b47baf-3103-4d30-953e-62628d4a795c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9"
Oct 30 23:23:57.140200 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:57.140105    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9"
Oct 30 23:23:57.156589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:57.156564    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:23:57.420551 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:57.420526    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pm8s9"]
Oct 30 23:23:57.422190 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:23:57.422160    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b47baf_3103_4d30_953e_62628d4a795c.slice/crio-53e5796f690cca482aac5729f6bc1bc349beb921f9b4bb54e13e073b7f4e7799 WatchSource:0}: Error finding container 53e5796f690cca482aac5729f6bc1bc349beb921f9b4bb54e13e073b7f4e7799: Status 404 returned error can't find the container with id 53e5796f690cca482aac5729f6bc1bc349beb921f9b4bb54e13e073b7f4e7799
Oct 30 23:23:57.446621 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:23:57.446550    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:23:57.446844 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:57.446821    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"]
Oct 30 23:23:57.448666 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:23:57.448641    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod06caa6ed_2a57_49ea_becf_ac4d8088e260.slice/crio-a551eac13267a769e6454444596c4a33ed90e8e266cdb0c8677661b0996ce47d WatchSource:0}: Error finding container a551eac13267a769e6454444596c4a33ed90e8e266cdb0c8677661b0996ce47d: Status 404 returned error can't find the container with id a551eac13267a769e6454444596c4a33ed90e8e266cdb0c8677661b0996ce47d
Oct 30 23:23:57.972288 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:57.972238    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z" event={"ID":"06caa6ed-2a57-49ea-becf-ac4d8088e260","Type":"ContainerStarted","Data":"a551eac13267a769e6454444596c4a33ed90e8e266cdb0c8677661b0996ce47d"}
Oct 30 23:23:57.973741 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:23:57.973708    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9" event={"ID":"78b47baf-3103-4d30-953e-62628d4a795c","Type":"ContainerStarted","Data":"53e5796f690cca482aac5729f6bc1bc349beb921f9b4bb54e13e073b7f4e7799"}
Oct 30 23:24:00.996645 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:24:00.996563    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z" event={"ID":"06caa6ed-2a57-49ea-becf-ac4d8088e260","Type":"ContainerStarted","Data":"83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4"}
Oct 30 23:24:00.997093 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:24:00.996729    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:24:01.018899 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:24:01.018842    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z" podStartSLOduration=1.8277955019999998 podStartE2EDuration="5.018828464s" totalImagesPullingTime="3.191032962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:01.016194132 +0000 UTC m=+299.942831713" watchObservedRunningTime="2026-10-30 23:24:01.018828464 +0000 UTC m=+299.945466044"
Oct 30 23:24:02.003469 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:24:02.003418    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9" event={"ID":"78b47baf-3103-4d30-953e-62628d4a795c","Type":"ContainerStarted","Data":"ac7a59b40d1a7fa3f3e3e22a8da56fa8e4364b46e61629592e5e91d26d87d638"}
Oct 30 23:24:02.003874 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:24:02.003677    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9"
Oct 30 23:24:02.025237 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:24:02.025186    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9" podStartSLOduration=2.286859624 podStartE2EDuration="6.025171372s" totalImagesPullingTime="3.738311748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:02.021727981 +0000 UTC m=+300.948365561" watchObservedRunningTime="2026-10-30 23:24:02.025171372 +0000 UTC m=+300.951808953"
Oct 30 23:24:27.445501 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:24:27.445421    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:24:32.005671 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:24:32.005635    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:24:33.014098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:24:33.014065    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pm8s9"
Oct 30 23:24:57.444970 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:24:57.444923    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:25:27.446108 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:25:27.446071    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:25:57.445424 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:25:57.445361    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:26:27.447012 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:26:27.446972    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:26:57.445782 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:26:57.445739    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:27:27.444652 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:27:27.444575    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:27:57.448592 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:27:57.448551    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:28:27.444839 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:28:27.444800    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:28:57.447665 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:28:57.447623    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:29:27.444994 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:29:27.444950    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:29:57.444008 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:29:57.443969    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:30:27.445157 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:30:27.445083    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:30:57.445459 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:30:57.445422    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:31:27.445898 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:31:27.445858    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:31:57.444879 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:31:57.444839    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:32:27.445104 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:32:27.445066    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:32:57.446240 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:32:57.446204    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:33:20.507097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:20.507013    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-q7bv9_02b04936-0271-4d0d-b14e-fc53c9b85bff/manager/0.log"
Oct 30 23:33:20.702873 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:20.702844    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"]
Oct 30 23:33:20.703095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:20.703061    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9" podUID="02b04936-0271-4d0d-b14e-fc53c9b85bff" containerName="manager" containerID="cri-o://4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10" gracePeriod=10
Oct 30 23:33:20.733660 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:20.733631    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"]
Oct 30 23:33:20.737123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:20.737106    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 30 23:33:20.751266 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:20.751239    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"]
Oct 30 23:33:20.927713 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:20.927680    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slw8p\" (UniqueName: \"kubernetes.io/projected/8b3411a0-327b-4919-8084-a07db4a5c4e8-kube-api-access-slw8p\") pod \"kuadrant-operator-controller-manager-678d87766f-lwfw7\" (UID: \"8b3411a0-327b-4919-8084-a07db4a5c4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 30 23:33:20.927870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:20.927730    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3411a0-327b-4919-8084-a07db4a5c4e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lwfw7\" (UID: \"8b3411a0-327b-4919-8084-a07db4a5c4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 30 23:33:20.965074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:20.965049    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:33:21.028202 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.028167    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3411a0-327b-4919-8084-a07db4a5c4e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lwfw7\" (UID: \"8b3411a0-327b-4919-8084-a07db4a5c4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 30 23:33:21.028411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.028264    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slw8p\" (UniqueName: \"kubernetes.io/projected/8b3411a0-327b-4919-8084-a07db4a5c4e8-kube-api-access-slw8p\") pod \"kuadrant-operator-controller-manager-678d87766f-lwfw7\" (UID: \"8b3411a0-327b-4919-8084-a07db4a5c4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 30 23:33:21.028637 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.028613    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3411a0-327b-4919-8084-a07db4a5c4e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lwfw7\" (UID: \"8b3411a0-327b-4919-8084-a07db4a5c4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 30 23:33:21.041250 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.041184    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slw8p\" (UniqueName: \"kubernetes.io/projected/8b3411a0-327b-4919-8084-a07db4a5c4e8-kube-api-access-slw8p\") pod \"kuadrant-operator-controller-manager-678d87766f-lwfw7\" (UID: \"8b3411a0-327b-4919-8084-a07db4a5c4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 30 23:33:21.095055 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.095018    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 30 23:33:21.129461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.129430    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b04936-0271-4d0d-b14e-fc53c9b85bff-kube-api-access-rpp2q\" (UniqueName: \"kubernetes.io/projected/02b04936-0271-4d0d-b14e-fc53c9b85bff-kube-api-access-rpp2q\") pod \"02b04936-0271-4d0d-b14e-fc53c9b85bff\" (UID: \"02b04936-0271-4d0d-b14e-fc53c9b85bff\") "
Oct 30 23:33:21.129623 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.129479    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b04936-0271-4d0d-b14e-fc53c9b85bff-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02b04936-0271-4d0d-b14e-fc53c9b85bff-extensions-socket-volume\") pod \"02b04936-0271-4d0d-b14e-fc53c9b85bff\" (UID: \"02b04936-0271-4d0d-b14e-fc53c9b85bff\") "
Oct 30 23:33:21.129898 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.129876    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b04936-0271-4d0d-b14e-fc53c9b85bff-extensions-socket-volume" pod "02b04936-0271-4d0d-b14e-fc53c9b85bff" (UID: "02b04936-0271-4d0d-b14e-fc53c9b85bff"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:33:21.131902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.131878    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b04936-0271-4d0d-b14e-fc53c9b85bff-kube-api-access-rpp2q" pod "02b04936-0271-4d0d-b14e-fc53c9b85bff" (UID: "02b04936-0271-4d0d-b14e-fc53c9b85bff"). InnerVolumeSpecName "kube-api-access-rpp2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:33:21.230791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.230757    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpp2q\" (UniqueName: \"kubernetes.io/projected/02b04936-0271-4d0d-b14e-fc53c9b85bff-kube-api-access-rpp2q\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:33:21.230791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.230784    4555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02b04936-0271-4d0d-b14e-fc53c9b85bff-extensions-socket-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:33:21.376682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.376602    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"]
Oct 30 23:33:21.379864 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:33:21.379841    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3411a0_327b_4919_8084_a07db4a5c4e8.slice/crio-ef7360e4f94e002764c5db76de9f62292c15b3b1a58e0f182af5b699b0a951a0 WatchSource:0}: Error finding container ef7360e4f94e002764c5db76de9f62292c15b3b1a58e0f182af5b699b0a951a0: Status 404 returned error can't find the container with id ef7360e4f94e002764c5db76de9f62292c15b3b1a58e0f182af5b699b0a951a0
Oct 30 23:33:21.687723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.687639    4555 generic.go:356] "Generic (PLEG): container finished" podID="02b04936-0271-4d0d-b14e-fc53c9b85bff" containerID="4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10" exitCode=0
Oct 30 23:33:21.687723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.687689    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"
Oct 30 23:33:21.688272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.687725    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9" event={"ID":"02b04936-0271-4d0d-b14e-fc53c9b85bff","Type":"ContainerDied","Data":"4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10"}
Oct 30 23:33:21.688272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.687791    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9" event={"ID":"02b04936-0271-4d0d-b14e-fc53c9b85bff","Type":"ContainerDied","Data":"c1fffa5b142b441c3c9fb00f67885a604481ae7413c2bd2dbe7671601d990d79"}
Oct 30 23:33:21.688272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.687818    4555 scope.go:122] "RemoveContainer" containerID="4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10"
Oct 30 23:33:21.689864 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.689829    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7" event={"ID":"8b3411a0-327b-4919-8084-a07db4a5c4e8","Type":"ContainerStarted","Data":"6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d"}
Oct 30 23:33:21.689999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.689880    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7" event={"ID":"8b3411a0-327b-4919-8084-a07db4a5c4e8","Type":"ContainerStarted","Data":"ef7360e4f94e002764c5db76de9f62292c15b3b1a58e0f182af5b699b0a951a0"}
Oct 30 23:33:21.689999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.689913    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 30 23:33:21.699184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.699160    4555 scope.go:122] "RemoveContainer" containerID="4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10"
Oct 30 23:33:21.699552 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:33:21.699523    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10\": container with ID starting with 4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10 not found: ID does not exist" containerID="4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10"
Oct 30 23:33:21.699651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.699562    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10"} err="failed to get container status \"4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10\": rpc error: code = NotFound desc = could not find container \"4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10\": container with ID starting with 4042af3b6849cb1209b5f45d5dc0b488704a02dbf0a573b03d1547eac2de3a10 not found: ID does not exist"
Oct 30 23:33:21.707630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.707607    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"]
Oct 30 23:33:21.712060 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.712038    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q7bv9"]
Oct 30 23:33:21.734213 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:21.734164    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7" podStartSLOduration=1.7341516110000001 podStartE2EDuration="1.734151611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:33:21.732668332 +0000 UTC m=+860.659305913" watchObservedRunningTime="2026-10-30 23:33:21.734151611 +0000 UTC m=+860.660789191"
Oct 30 23:33:23.205018 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:23.204987    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b04936-0271-4d0d-b14e-fc53c9b85bff" path="/var/lib/kubelet/pods/02b04936-0271-4d0d-b14e-fc53c9b85bff/volumes"
Oct 30 23:33:27.445485 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:33:27.445446    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:33:32.698133 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:32.698101    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 30 23:33:41.197805 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.197759    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"]
Oct 30 23:33:41.198978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.198237    4555 cpu_manager.go:414] "containerMap: removing container" podUID="02b04936-0271-4d0d-b14e-fc53c9b85bff" containerName="manager"
Oct 30 23:33:41.198978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.198252    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b04936-0271-4d0d-b14e-fc53c9b85bff" containerName="manager"
Oct 30 23:33:41.198978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.198456    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b04936-0271-4d0d-b14e-fc53c9b85bff" containerName="manager"
Oct 30 23:33:41.202942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.202916    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.214005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.213978    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"]
Oct 30 23:33:41.225186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.219421    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 23:33:41.225186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.219555    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 23:33:41.225186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.219665    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jzg7x\""
Oct 30 23:33:41.283806 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.283777    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87feb72-91e5-456e-8035-68fed3fa7583-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wfl4g\" (UID: \"f87feb72-91e5-456e-8035-68fed3fa7583\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.283987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.283814    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f87feb72-91e5-456e-8035-68fed3fa7583-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wfl4g\" (UID: \"f87feb72-91e5-456e-8035-68fed3fa7583\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.283987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.283839    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8hxx\" (UniqueName: \"kubernetes.io/projected/f87feb72-91e5-456e-8035-68fed3fa7583-kube-api-access-w8hxx\") pod \"kuadrant-console-plugin-66c4fb575d-wfl4g\" (UID: \"f87feb72-91e5-456e-8035-68fed3fa7583\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.384568 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.384528    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87feb72-91e5-456e-8035-68fed3fa7583-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wfl4g\" (UID: \"f87feb72-91e5-456e-8035-68fed3fa7583\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.384773 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.384577    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f87feb72-91e5-456e-8035-68fed3fa7583-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wfl4g\" (UID: \"f87feb72-91e5-456e-8035-68fed3fa7583\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.384773 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.384602    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8hxx\" (UniqueName: \"kubernetes.io/projected/f87feb72-91e5-456e-8035-68fed3fa7583-kube-api-access-w8hxx\") pod \"kuadrant-console-plugin-66c4fb575d-wfl4g\" (UID: \"f87feb72-91e5-456e-8035-68fed3fa7583\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.385532 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.385504    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f87feb72-91e5-456e-8035-68fed3fa7583-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wfl4g\" (UID: \"f87feb72-91e5-456e-8035-68fed3fa7583\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.387544 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.387525    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87feb72-91e5-456e-8035-68fed3fa7583-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wfl4g\" (UID: \"f87feb72-91e5-456e-8035-68fed3fa7583\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.398099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.398072    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8hxx\" (UniqueName: \"kubernetes.io/projected/f87feb72-91e5-456e-8035-68fed3fa7583-kube-api-access-w8hxx\") pod \"kuadrant-console-plugin-66c4fb575d-wfl4g\" (UID: \"f87feb72-91e5-456e-8035-68fed3fa7583\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.533122 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.533093    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"
Oct 30 23:33:41.586661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.586626    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:33:41.591901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.590886    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b"
Oct 30 23:33:41.599131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.599090    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:33:41.601088 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.601054    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 23:33:41.631519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.631484    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:33:41.686705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.686671    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdd72\" (UniqueName: \"kubernetes.io/projected/6f402bfd-be3a-4b13-b402-4e6b75dfe2bd-kube-api-access-fdd72\") pod \"limitador-limitador-7c9b87d574-9w98b\" (UID: \"6f402bfd-be3a-4b13-b402-4e6b75dfe2bd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b"
Oct 30 23:33:41.686873 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.686814    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6f402bfd-be3a-4b13-b402-4e6b75dfe2bd-config-file\") pod \"limitador-limitador-7c9b87d574-9w98b\" (UID: \"6f402bfd-be3a-4b13-b402-4e6b75dfe2bd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b"
Oct 30 23:33:41.788080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.787990    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6f402bfd-be3a-4b13-b402-4e6b75dfe2bd-config-file\") pod \"limitador-limitador-7c9b87d574-9w98b\" (UID: \"6f402bfd-be3a-4b13-b402-4e6b75dfe2bd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b"
Oct 30 23:33:41.788080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.788025    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd72\" (UniqueName: \"kubernetes.io/projected/6f402bfd-be3a-4b13-b402-4e6b75dfe2bd-kube-api-access-fdd72\") pod \"limitador-limitador-7c9b87d574-9w98b\" (UID: \"6f402bfd-be3a-4b13-b402-4e6b75dfe2bd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b"
Oct 30 23:33:41.788720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.788700    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6f402bfd-be3a-4b13-b402-4e6b75dfe2bd-config-file\") pod \"limitador-limitador-7c9b87d574-9w98b\" (UID: \"6f402bfd-be3a-4b13-b402-4e6b75dfe2bd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b"
Oct 30 23:33:41.800640 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.800612    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdd72\" (UniqueName: \"kubernetes.io/projected/6f402bfd-be3a-4b13-b402-4e6b75dfe2bd-kube-api-access-fdd72\") pod \"limitador-limitador-7c9b87d574-9w98b\" (UID: \"6f402bfd-be3a-4b13-b402-4e6b75dfe2bd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b"
Oct 30 23:33:41.815480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.815432    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g"]
Oct 30 23:33:41.817709 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:33:41.817682    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf87feb72_91e5_456e_8035_68fed3fa7583.slice/crio-6d9138580b7f4a70530fa34e7e879f214545786d149c8916465e05b941a30dd7 WatchSource:0}: Error finding container 6d9138580b7f4a70530fa34e7e879f214545786d149c8916465e05b941a30dd7: Status 404 returned error can't find the container with id 6d9138580b7f4a70530fa34e7e879f214545786d149c8916465e05b941a30dd7
Oct 30 23:33:41.819058 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.819042    4555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:33:41.909992 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:41.909957    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b"
Oct 30 23:33:42.194900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:42.194878    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:33:42.197294 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:33:42.197264    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f402bfd_be3a_4b13_b402_4e6b75dfe2bd.slice/crio-eb48cf361c194c71fc94211650c1581d101abb2a931f6f4fd2a299a3fb3ff298 WatchSource:0}: Error finding container eb48cf361c194c71fc94211650c1581d101abb2a931f6f4fd2a299a3fb3ff298: Status 404 returned error can't find the container with id eb48cf361c194c71fc94211650c1581d101abb2a931f6f4fd2a299a3fb3ff298
Oct 30 23:33:42.796074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:42.796041    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g" event={"ID":"f87feb72-91e5-456e-8035-68fed3fa7583","Type":"ContainerStarted","Data":"6d9138580b7f4a70530fa34e7e879f214545786d149c8916465e05b941a30dd7"}
Oct 30 23:33:42.798517 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:42.798485    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b" event={"ID":"6f402bfd-be3a-4b13-b402-4e6b75dfe2bd","Type":"ContainerStarted","Data":"eb48cf361c194c71fc94211650c1581d101abb2a931f6f4fd2a299a3fb3ff298"}
Oct 30 23:33:44.810488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:44.810452    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b" event={"ID":"6f402bfd-be3a-4b13-b402-4e6b75dfe2bd","Type":"ContainerStarted","Data":"ff34301979ce54612626e3d67532ae2d32da317f03b6e40aebbc92ab874afd19"}
Oct 30 23:33:44.810953 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:44.810611    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b"
Oct 30 23:33:44.831420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:44.831346    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b" podStartSLOduration=1.745387668 podStartE2EDuration="3.831332059s" totalImagesPullingTime="2.085944391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:33:44.828896548 +0000 UTC m=+883.755534130" watchObservedRunningTime="2026-10-30 23:33:44.831332059 +0000 UTC m=+883.757969639"
Oct 30 23:33:47.826833 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:47.826793    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g" event={"ID":"f87feb72-91e5-456e-8035-68fed3fa7583","Type":"ContainerStarted","Data":"97e1145ea861b58b22360fc07605b1a1ab45f4c446fb99c21f6ca216ab7e9392"}
Oct 30 23:33:47.846308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:47.846257    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wfl4g" podStartSLOduration=1.6715176889999999 podStartE2EDuration="6.846244227s" totalImagesPullingTime="5.174726538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:33:47.844173676 +0000 UTC m=+886.770811259" watchObservedRunningTime="2026-10-30 23:33:47.846244227 +0000 UTC m=+886.772881808"
Oct 30 23:33:55.816737 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:33:55.816707    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9w98b"
Oct 30 23:33:57.445482 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:33:57.445448    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:34:27.445797 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:34:27.445757    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:34:57.446383 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:34:57.446327    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:34:57.967963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:34:57.967926    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:35:07.204587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:07.204549    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-mjk4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:35:07.205948 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:07.204620    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" podUID="ea819a48-154f-4c7a-a747-3306ac423289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:35:08.236233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:08.236198    4555 generic.go:356] "Generic (PLEG): container finished" podID="ea819a48-154f-4c7a-a747-3306ac423289" containerID="354d7b69d7bbf5ada5a45d5aff2724c410025e092dc2e590ff84b790d9036a03" exitCode=0
Oct 30 23:35:08.236644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:08.236261    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" event={"ID":"ea819a48-154f-4c7a-a747-3306ac423289","Type":"ContainerDied","Data":"354d7b69d7bbf5ada5a45d5aff2724c410025e092dc2e590ff84b790d9036a03"}
Oct 30 23:35:08.236644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:08.236621    4555 scope.go:122] "RemoveContainer" containerID="354d7b69d7bbf5ada5a45d5aff2724c410025e092dc2e590ff84b790d9036a03"
Oct 30 23:35:08.237762 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:08.237732    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:35:08.237817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:08.237803    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:35:08.237867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:08.237856    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:35:09.048868 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.048835    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"]
Oct 30 23:35:09.051283 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.051263    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.094351 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.094321    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"]
Oct 30 23:35:09.130494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.130453    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcs89\" (UniqueName: \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-kube-api-access-hcs89\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.130494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.130497    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.130731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.130608    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.130731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.130636    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.130824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.130727    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.130824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.130759    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.130824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.130787    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.130972 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.130850    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.130972 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.130910    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c69a343e-ca93-47a1-892b-c17cb341cff0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.232325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.232284    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.232325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.232324    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.232636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.232404    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.232636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.232435    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.232636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.232475    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.232636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.232507    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.232636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.232534    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c69a343e-ca93-47a1-892b-c17cb341cff0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.232636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.232577    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcs89\" (UniqueName: \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-kube-api-access-hcs89\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.232636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.232607    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.233232 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.233203    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c69a343e-ca93-47a1-892b-c17cb341cff0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.233395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.233348    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.233498 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.233472    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.233498 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.233484    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.233626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.233603    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.235282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.235262    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.235447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.235430    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.245007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.244963    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" event={"ID":"ea819a48-154f-4c7a-a747-3306ac423289","Type":"ContainerStarted","Data":"6c7d29b1506c83ded239f8ad37dd90cd8ed46512b25c2be19a68aefc267ffe7c"}
Oct 30 23:35:09.246183 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.246155    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.246294 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.246216    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcs89\" (UniqueName: \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-kube-api-access-hcs89\") pod \"maas-default-gateway-openshift-default-5df48fcc99-cwtmx\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.393814 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.393720    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:09.691340 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.691249    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"]
Oct 30 23:35:09.693558 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:35:09.693523    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69a343e_ca93_47a1_892b_c17cb341cff0.slice/crio-3ed2a60ef76ca7e3f918ab7c779d00e192ddeb25ad217c84798ed2a79225eabc WatchSource:0}: Error finding container 3ed2a60ef76ca7e3f918ab7c779d00e192ddeb25ad217c84798ed2a79225eabc: Status 404 returned error can't find the container with id 3ed2a60ef76ca7e3f918ab7c779d00e192ddeb25ad217c84798ed2a79225eabc
Oct 30 23:35:09.695831 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.695800    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:35:09.695906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.695869    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:35:09.695906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:09.695898    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:35:10.169397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.165773    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:35:10.169397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.165914    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-mjk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 23:35:10.169397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.165971    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" podUID="ea819a48-154f-4c7a-a747-3306ac423289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 23:35:10.251407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.251348    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" event={"ID":"c69a343e-ca93-47a1-892b-c17cb341cff0","Type":"ContainerStarted","Data":"73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336"}
Oct 30 23:35:10.251876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.251429    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" event={"ID":"c69a343e-ca93-47a1-892b-c17cb341cff0","Type":"ContainerStarted","Data":"3ed2a60ef76ca7e3f918ab7c779d00e192ddeb25ad217c84798ed2a79225eabc"}
Oct 30 23:35:10.293395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.290443    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" podStartSLOduration=1.290421747 podStartE2EDuration="1.290421747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:35:10.288961459 +0000 UTC m=+969.215599045" watchObservedRunningTime="2026-10-30 23:35:10.290421747 +0000 UTC m=+969.217059329"
Oct 30 23:35:10.396333 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.396297    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:10.396776 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.396749    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-cwtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 23:35:10.396990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.396955    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 23:35:10.913165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.913118    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"]
Oct 30 23:35:10.915800 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.915773    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"
Oct 30 23:35:10.923259 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.923233    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"]
Oct 30 23:35:10.949727 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.949698    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qzr4\" (UniqueName: \"kubernetes.io/projected/d4e50936-fbfc-49ab-b816-30d80f7dc3cb-kube-api-access-8qzr4\") pod \"llmisvc-controller-manager-6466c554d8-qpd89\" (UID: \"d4e50936-fbfc-49ab-b816-30d80f7dc3cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"
Oct 30 23:35:10.949978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:10.949954    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4e50936-fbfc-49ab-b816-30d80f7dc3cb-cert\") pod \"llmisvc-controller-manager-6466c554d8-qpd89\" (UID: \"d4e50936-fbfc-49ab-b816-30d80f7dc3cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"
Oct 30 23:35:11.050513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:11.050459    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4e50936-fbfc-49ab-b816-30d80f7dc3cb-cert\") pod \"llmisvc-controller-manager-6466c554d8-qpd89\" (UID: \"d4e50936-fbfc-49ab-b816-30d80f7dc3cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"
Oct 30 23:35:11.050709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:11.050549    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qzr4\" (UniqueName: \"kubernetes.io/projected/d4e50936-fbfc-49ab-b816-30d80f7dc3cb-kube-api-access-8qzr4\") pod \"llmisvc-controller-manager-6466c554d8-qpd89\" (UID: \"d4e50936-fbfc-49ab-b816-30d80f7dc3cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"
Oct 30 23:35:11.053062 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:11.053033    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4e50936-fbfc-49ab-b816-30d80f7dc3cb-cert\") pod \"llmisvc-controller-manager-6466c554d8-qpd89\" (UID: \"d4e50936-fbfc-49ab-b816-30d80f7dc3cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"
Oct 30 23:35:11.090397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:11.090325    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qzr4\" (UniqueName: \"kubernetes.io/projected/d4e50936-fbfc-49ab-b816-30d80f7dc3cb-kube-api-access-8qzr4\") pod \"llmisvc-controller-manager-6466c554d8-qpd89\" (UID: \"d4e50936-fbfc-49ab-b816-30d80f7dc3cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"
Oct 30 23:35:11.168456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:11.164811    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-mjk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 23:35:11.168456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:11.164874    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" podUID="ea819a48-154f-4c7a-a747-3306ac423289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 23:35:11.230598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:11.230563    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"
Oct 30 23:35:11.394848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:11.394808    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-cwtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 23:35:11.399526 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:11.399481    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 23:35:11.549307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:11.549059    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"]
Oct 30 23:35:11.551867 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:35:11.551815    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e50936_fbfc_49ab_b816_30d80f7dc3cb.slice/crio-44d1a4d426de6fb891f6157dbfc7e4baaa8cdd1ed7bbee5a87986961344fcc3d WatchSource:0}: Error finding container 44d1a4d426de6fb891f6157dbfc7e4baaa8cdd1ed7bbee5a87986961344fcc3d: Status 404 returned error can't find the container with id 44d1a4d426de6fb891f6157dbfc7e4baaa8cdd1ed7bbee5a87986961344fcc3d
Oct 30 23:35:12.166274 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:12.166232    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-mjk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 23:35:12.166573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:12.166548    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" podUID="ea819a48-154f-4c7a-a747-3306ac423289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 23:35:12.264580 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:12.264456    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89" event={"ID":"d4e50936-fbfc-49ab-b816-30d80f7dc3cb","Type":"ContainerStarted","Data":"9ac740756a9825178e2893cc2aa7b5fa4b2fc3e27944af224e202c81bfb46f5a"}
Oct 30 23:35:12.264846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:12.264812    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"
Oct 30 23:35:12.265016 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:12.264997    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89" event={"ID":"d4e50936-fbfc-49ab-b816-30d80f7dc3cb","Type":"ContainerStarted","Data":"44d1a4d426de6fb891f6157dbfc7e4baaa8cdd1ed7bbee5a87986961344fcc3d"}
Oct 30 23:35:12.292878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:12.291159    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89" podStartSLOduration=1.897088009 podStartE2EDuration="2.291122983s" totalImagesPullingTime="394.034974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:35:12.288970953 +0000 UTC m=+971.215608535" watchObservedRunningTime="2026-10-30 23:35:12.291122983 +0000 UTC m=+971.217760554"
Oct 30 23:35:12.394835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:12.394798    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-cwtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 23:35:12.395111 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:12.395082    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 23:35:13.167143 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:13.166668    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-mjk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 23:35:13.167143 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:13.166734    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" podUID="ea819a48-154f-4c7a-a747-3306ac423289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 23:35:13.395398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:13.394984    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-cwtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 23:35:13.395398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:13.395068    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 23:35:14.170401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:14.165871    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-mjk4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 23:35:14.170401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:14.165938    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" podUID="ea819a48-154f-4c7a-a747-3306ac423289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 23:35:14.394155 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:14.394114    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-cwtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 23:35:14.394327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:14.394190    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 23:35:15.186976 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:15.186944    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:35:15.187495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:15.187322    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:35:15.188455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:15.188432    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b"
Oct 30 23:35:15.394401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:15.394340    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-cwtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 23:35:15.394585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:15.394431    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 23:35:16.398887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:16.398855    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:16.399252 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:16.399141    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:16.399780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:16.399765    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:35:27.445350 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:35:27.445310    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:35:44.310347 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:44.310312    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qpd89"
Oct 30 23:35:44.366563 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:44.366530    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"]
Oct 30 23:35:44.367029 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:44.366996    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z" podUID="06caa6ed-2a57-49ea-becf-ac4d8088e260" containerName="manager" containerID="cri-o://83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4" gracePeriod=30
Oct 30 23:35:44.626041 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:44.626016    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:35:44.677450 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:44.677417    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06caa6ed-2a57-49ea-becf-ac4d8088e260-kube-api-access-j45nr\" (UniqueName: \"kubernetes.io/projected/06caa6ed-2a57-49ea-becf-ac4d8088e260-kube-api-access-j45nr\") pod \"06caa6ed-2a57-49ea-becf-ac4d8088e260\" (UID: \"06caa6ed-2a57-49ea-becf-ac4d8088e260\") "
Oct 30 23:35:44.677616 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:44.677511    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06caa6ed-2a57-49ea-becf-ac4d8088e260-cert\" (UniqueName: \"kubernetes.io/secret/06caa6ed-2a57-49ea-becf-ac4d8088e260-cert\") pod \"06caa6ed-2a57-49ea-becf-ac4d8088e260\" (UID: \"06caa6ed-2a57-49ea-becf-ac4d8088e260\") "
Oct 30 23:35:44.679759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:44.679720    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06caa6ed-2a57-49ea-becf-ac4d8088e260-cert" pod "06caa6ed-2a57-49ea-becf-ac4d8088e260" (UID: "06caa6ed-2a57-49ea-becf-ac4d8088e260"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:35:44.679889 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:44.679833    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06caa6ed-2a57-49ea-becf-ac4d8088e260-kube-api-access-j45nr" pod "06caa6ed-2a57-49ea-becf-ac4d8088e260" (UID: "06caa6ed-2a57-49ea-becf-ac4d8088e260"). InnerVolumeSpecName "kube-api-access-j45nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:44.778585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:44.778543    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j45nr\" (UniqueName: \"kubernetes.io/projected/06caa6ed-2a57-49ea-becf-ac4d8088e260-kube-api-access-j45nr\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:35:44.778585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:44.778578    4555 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06caa6ed-2a57-49ea-becf-ac4d8088e260-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:35:45.463104 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:45.463019    4555 generic.go:356] "Generic (PLEG): container finished" podID="06caa6ed-2a57-49ea-becf-ac4d8088e260" containerID="83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4" exitCode=0
Oct 30 23:35:45.463104 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:45.463078    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"
Oct 30 23:35:45.463104 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:45.463082    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z" event={"ID":"06caa6ed-2a57-49ea-becf-ac4d8088e260","Type":"ContainerDied","Data":"83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4"}
Oct 30 23:35:45.463682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:45.463138    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z" event={"ID":"06caa6ed-2a57-49ea-becf-ac4d8088e260","Type":"ContainerDied","Data":"a551eac13267a769e6454444596c4a33ed90e8e266cdb0c8677661b0996ce47d"}
Oct 30 23:35:45.463682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:45.463166    4555 scope.go:122] "RemoveContainer" containerID="83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4"
Oct 30 23:35:45.472275 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:45.472254    4555 scope.go:122] "RemoveContainer" containerID="83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4"
Oct 30 23:35:45.472570 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:35:45.472552    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4\": container with ID starting with 83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4 not found: ID does not exist" containerID="83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4"
Oct 30 23:35:45.472617 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:45.472579    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4"} err="failed to get container status \"83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4\": rpc error: code = NotFound desc = could not find container \"83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4\": container with ID starting with 83544f95c6717519f68d0d130e3d1d1b3f21661fda290ed33271703fe203c6a4 not found: ID does not exist"
Oct 30 23:35:45.480764 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:45.480738    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"]
Oct 30 23:35:45.486140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:45.486113    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-46r5z"]
Oct 30 23:35:47.204916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:35:47.204888    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06caa6ed-2a57-49ea-becf-ac4d8088e260" path="/var/lib/kubelet/pods/06caa6ed-2a57-49ea-becf-ac4d8088e260/volumes"
Oct 30 23:35:57.444992 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:35:57.444944    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:36:02.402734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:02.402700    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:36:05.112170 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:05.112132    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:36:06.055818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:06.055784    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:36:09.109943 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.109899    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"]
Oct 30 23:36:09.110494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.110479    4555 cpu_manager.go:414] "containerMap: removing container" podUID="06caa6ed-2a57-49ea-becf-ac4d8088e260" containerName="manager"
Oct 30 23:36:09.110550 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.110497    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06caa6ed-2a57-49ea-becf-ac4d8088e260" containerName="manager"
Oct 30 23:36:09.110595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.110584    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="06caa6ed-2a57-49ea-becf-ac4d8088e260" containerName="manager"
Oct 30 23:36:09.115715 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.115680    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.128237 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.127225    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"]
Oct 30 23:36:09.202478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.202445    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.202651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.202584    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.202651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.202641    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.202725 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.202694    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.202757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.202726    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbgsp\" (UniqueName: \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-kube-api-access-rbgsp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.202794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.202758    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.202794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.202782    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.202864 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.202825    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.202897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.202857    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.304205 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.304152    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.304445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.304213    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbgsp\" (UniqueName: \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-kube-api-access-rbgsp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.304445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.304354    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.304445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.304407    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.304651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.304450    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.304651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.304489    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.304651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.304553    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.304651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.304602    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.304651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.304648    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.305057 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.305033    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.305176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.305149    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.305495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.305468    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.305657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.305632    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.306178 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.306152    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.320925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.320828    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.321102 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.320996    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.325861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.325827    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.326422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.326361    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbgsp\" (UniqueName: \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-kube-api-access-rbgsp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2jkbv\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.436079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.435890    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:09.743574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.743549    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"]
Oct 30 23:36:09.746861 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:36:09.746832    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f007baf_e820_44b9_b1ba_804cbd0fcbe3.slice/crio-5549daa0904dda13656d632aa151188b28a16646852b93e0d658b441939f77b1 WatchSource:0}: Error finding container 5549daa0904dda13656d632aa151188b28a16646852b93e0d658b441939f77b1: Status 404 returned error can't find the container with id 5549daa0904dda13656d632aa151188b28a16646852b93e0d658b441939f77b1
Oct 30 23:36:09.749597 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.749556    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:36:09.749682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.749650    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:36:09.749720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:09.749698    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:36:10.615902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:10.615858    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" event={"ID":"6f007baf-e820-44b9-b1ba-804cbd0fcbe3","Type":"ContainerStarted","Data":"9f9b508ca6200605b0e24d661be564221909cb46a8e5e647c31a3c9332b69b44"}
Oct 30 23:36:10.621610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:10.621572    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" event={"ID":"6f007baf-e820-44b9-b1ba-804cbd0fcbe3","Type":"ContainerStarted","Data":"5549daa0904dda13656d632aa151188b28a16646852b93e0d658b441939f77b1"}
Oct 30 23:36:10.660395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:10.657505    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" podStartSLOduration=1.6574849440000001 podStartE2EDuration="1.657484944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:36:10.655859038 +0000 UTC m=+1029.582496646" watchObservedRunningTime="2026-10-30 23:36:10.657484944 +0000 UTC m=+1029.584122525"
Oct 30 23:36:11.323450 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:11.323217    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:36:11.438179 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:11.438147    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:11.438393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:11.438347    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2jkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 23:36:11.438475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:11.438410    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 23:36:12.437357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:12.437319    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2jkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 23:36:12.437775 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:12.437408    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 23:36:13.437112 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:13.437078    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2jkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 23:36:13.437339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:13.437140    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 23:36:14.437955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:14.437915    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2jkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 23:36:14.438628 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:14.438586    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 23:36:15.437834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:15.437794    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2jkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 23:36:15.438312 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:15.438272    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 23:36:16.436960 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:16.436918    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2jkbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 23:36:16.437185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:16.436987    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 23:36:16.571577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:16.571535    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:36:17.991616 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:17.991573    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:17.993564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:17.993526    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:17.995743 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:36:17.995713    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:36:27.446646 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:36:27.446606    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:36:57.445444 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:36:57.445406    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:37:27.447680 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:37:27.447635    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:37:31.424462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.424427    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"]
Oct 30 23:37:31.429816 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.429780    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:37:31.441318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.441289    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"]
Oct 30 23:37:31.481672 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.481641    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-611ef3-dockercfg-m9bvp\""
Oct 30 23:37:31.483519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.483497    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-611ef3\""
Oct 30 23:37:31.624836 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.624803    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm4zm\" (UniqueName: \"kubernetes.io/projected/8b131cf2-9074-493b-a5c9-0ee97d47a566-kube-api-access-zm4zm\") pod \"payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd\" (UID: \"8b131cf2-9074-493b-a5c9-0ee97d47a566\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:37:31.624836 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.624842    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b131cf2-9074-493b-a5c9-0ee97d47a566-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd\" (UID: \"8b131cf2-9074-493b-a5c9-0ee97d47a566\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:37:31.725436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.725329    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm4zm\" (UniqueName: \"kubernetes.io/projected/8b131cf2-9074-493b-a5c9-0ee97d47a566-kube-api-access-zm4zm\") pod \"payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd\" (UID: \"8b131cf2-9074-493b-a5c9-0ee97d47a566\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:37:31.725436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.725382    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b131cf2-9074-493b-a5c9-0ee97d47a566-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd\" (UID: \"8b131cf2-9074-493b-a5c9-0ee97d47a566\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:37:31.726085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.726066    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b131cf2-9074-493b-a5c9-0ee97d47a566-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd\" (UID: \"8b131cf2-9074-493b-a5c9-0ee97d47a566\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:37:31.740843 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.740810    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm4zm\" (UniqueName: \"kubernetes.io/projected/8b131cf2-9074-493b-a5c9-0ee97d47a566-kube-api-access-zm4zm\") pod \"payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd\" (UID: \"8b131cf2-9074-493b-a5c9-0ee97d47a566\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:37:31.745667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:31.745651    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:37:32.035089 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.035059    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"]
Oct 30 23:37:32.092097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.092058    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd" event={"ID":"8b131cf2-9074-493b-a5c9-0ee97d47a566","Type":"ContainerStarted","Data":"c24d1db942803c77a8c4cea953c14dcdddd83d3b82b806f12b45e7fe5e9721a3"}
Oct 30 23:37:32.727486 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.727454    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"]
Oct 30 23:37:32.732176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.732149    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.754467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.754429    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"]
Oct 30 23:37:32.789212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.789178    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-611ef3-serving-cert\""
Oct 30 23:37:32.789448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.789256    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:37:32.789448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.789258    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-59scr\""
Oct 30 23:37:32.789448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.789339    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:37:32.789448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.789345    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 23:37:32.835914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.835875    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.836185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.835931    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2g6q\" (UniqueName: \"kubernetes.io/projected/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-kube-api-access-r2g6q\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.836185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.836083    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.836185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.836131    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-metrics-certs\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.937062 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.937023    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.937278 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.937093    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-metrics-certs\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.937278 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.937153    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.937278 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.937188    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2g6q\" (UniqueName: \"kubernetes.io/projected/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-kube-api-access-r2g6q\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.937676 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:37:32.937656    4555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-serving-cert: secret "maas-api-e2e-worker-w1-611ef3-serving-cert" not found
Oct 30 23:37:32.937781 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:37:32.937719    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-maas-api-tls podName:73c8bb83-c123-421e-b1ab-7a9fbd44c9f9 nodeName:}" failed. No retries permitted until 2026-10-30 23:37:33.437701089 +0000 UTC m=+1112.364338651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-maas-api-tls") pod "maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc" (UID: "73c8bb83-c123-421e-b1ab-7a9fbd44c9f9") : secret "maas-api-e2e-worker-w1-611ef3-serving-cert" not found
Oct 30 23:37:32.941068 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.938825    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.942424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.941789    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-metrics-certs\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:32.960230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:32.960196    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2g6q\" (UniqueName: \"kubernetes.io/projected/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-kube-api-access-r2g6q\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:33.442271 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.442236    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:33.446766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.446734    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:33.658156 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.658122    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:33.815708 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.815654    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"]
Oct 30 23:37:33.822706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.822683    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 30 23:37:33.832773 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.831966    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-9ef03a-dockercfg-56bx9\""
Oct 30 23:37:33.835754 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.835724    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"]
Oct 30 23:37:33.838580 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.838556    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9ef03a\""
Oct 30 23:37:33.947532 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.947492    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skbmj\" (UniqueName: \"kubernetes.io/projected/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-kube-api-access-skbmj\") pod \"payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk\" (UID: \"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 30 23:37:33.947730 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.947588    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk\" (UID: \"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 30 23:37:33.960757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:33.959957    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"]
Oct 30 23:37:33.976439 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:37:33.976399    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c8bb83_c123_421e_b1ab_7a9fbd44c9f9.slice/crio-966455bd055025c9be200b769a1c95965efb5e4cffeb16dcd84dc2e692956703 WatchSource:0}: Error finding container 966455bd055025c9be200b769a1c95965efb5e4cffeb16dcd84dc2e692956703: Status 404 returned error can't find the container with id 966455bd055025c9be200b769a1c95965efb5e4cffeb16dcd84dc2e692956703
Oct 30 23:37:34.048051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.048015    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk\" (UID: \"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 30 23:37:34.048240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.048103    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skbmj\" (UniqueName: \"kubernetes.io/projected/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-kube-api-access-skbmj\") pod \"payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk\" (UID: \"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 30 23:37:34.048870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.048846    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk\" (UID: \"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 30 23:37:34.060601 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.060579    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skbmj\" (UniqueName: \"kubernetes.io/projected/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-kube-api-access-skbmj\") pod \"payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk\" (UID: \"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 30 23:37:34.108157 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.108120    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc" event={"ID":"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9","Type":"ContainerStarted","Data":"966455bd055025c9be200b769a1c95965efb5e4cffeb16dcd84dc2e692956703"}
Oct 30 23:37:34.148949 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.148910    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 30 23:37:34.445547 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.445519    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"]
Oct 30 23:37:34.447552 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:37:34.447522    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0e6ddc_4637_46b7_b4e6_aaec3d2182c6.slice/crio-4dd48eeccc34be522d7d197df49b79b993b36ac4e6bd43342a470c888491de10 WatchSource:0}: Error finding container 4dd48eeccc34be522d7d197df49b79b993b36ac4e6bd43342a470c888491de10: Status 404 returned error can't find the container with id 4dd48eeccc34be522d7d197df49b79b993b36ac4e6bd43342a470c888491de10
Oct 30 23:37:34.671126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.671084    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:37:34.724911 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.724434    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"]
Oct 30 23:37:34.728809 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.728766    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 30 23:37:34.739214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.739187    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b567bb\""
Oct 30 23:37:34.739336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.739258    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b567bb-dockercfg-g2mjl\""
Oct 30 23:37:34.753474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.753446    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"]
Oct 30 23:37:34.759657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.758627    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsjx\" (UniqueName: \"kubernetes.io/projected/4accfd35-d3c1-4b35-8271-43ffc726a5d3-kube-api-access-tjsjx\") pod \"payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb\" (UID: \"4accfd35-d3c1-4b35-8271-43ffc726a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 30 23:37:34.759657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.758768    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4accfd35-d3c1-4b35-8271-43ffc726a5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb\" (UID: \"4accfd35-d3c1-4b35-8271-43ffc726a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 30 23:37:34.860246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.860211    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsjx\" (UniqueName: \"kubernetes.io/projected/4accfd35-d3c1-4b35-8271-43ffc726a5d3-kube-api-access-tjsjx\") pod \"payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb\" (UID: \"4accfd35-d3c1-4b35-8271-43ffc726a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 30 23:37:34.860697 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.860294    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4accfd35-d3c1-4b35-8271-43ffc726a5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb\" (UID: \"4accfd35-d3c1-4b35-8271-43ffc726a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 30 23:37:34.860955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.860935    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4accfd35-d3c1-4b35-8271-43ffc726a5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb\" (UID: \"4accfd35-d3c1-4b35-8271-43ffc726a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 30 23:37:34.881823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:34.881764    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsjx\" (UniqueName: \"kubernetes.io/projected/4accfd35-d3c1-4b35-8271-43ffc726a5d3-kube-api-access-tjsjx\") pod \"payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb\" (UID: \"4accfd35-d3c1-4b35-8271-43ffc726a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 30 23:37:35.049635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:35.046397    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 30 23:37:35.117887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:35.117849    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk" event={"ID":"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6","Type":"ContainerStarted","Data":"04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6"}
Oct 30 23:37:35.118013 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:35.117958    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk" event={"ID":"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6","Type":"ContainerStarted","Data":"4dd48eeccc34be522d7d197df49b79b993b36ac4e6bd43342a470c888491de10"}
Oct 30 23:37:35.118865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:35.118823    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 30 23:37:35.122723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:35.122694    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd" event={"ID":"8b131cf2-9074-493b-a5c9-0ee97d47a566","Type":"ContainerStarted","Data":"ad6991788b8e4184f3c0a2292586b6c53ad8c342409a8e354cc1c2cfba8839ea"}
Oct 30 23:37:35.122971 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:35.122952    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:37:35.141172 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:35.141118    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk" podStartSLOduration=1.66898129 podStartE2EDuration="2.141097389s" totalImagesPullingTime="472.116099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:35.13668723 +0000 UTC m=+1114.063324815" watchObservedRunningTime="2026-10-30 23:37:35.141097389 +0000 UTC m=+1114.067734971"
Oct 30 23:37:35.444355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:35.444264    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd" podStartSLOduration=1.7437885309999999 podStartE2EDuration="4.444243056s" totalImagesPullingTime="2.700454525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:35.159102329 +0000 UTC m=+1114.085739911" watchObservedRunningTime="2026-10-30 23:37:35.444243056 +0000 UTC m=+1114.370880639"
Oct 30 23:37:35.444587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:35.444573    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"]
Oct 30 23:37:36.133584 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:36.133543    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb" event={"ID":"4accfd35-d3c1-4b35-8271-43ffc726a5d3","Type":"ContainerStarted","Data":"32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc"}
Oct 30 23:37:36.133986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:36.133603    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb" event={"ID":"4accfd35-d3c1-4b35-8271-43ffc726a5d3","Type":"ContainerStarted","Data":"d4b9900b3378ba949b28eba05d19e732312a0bbe52a4bd4d21934566678423c7"}
Oct 30 23:37:36.156610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:36.156560    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb" podStartSLOduration=2.15654376 podStartE2EDuration="2.15654376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:36.154444216 +0000 UTC m=+1115.081081791" watchObservedRunningTime="2026-10-30 23:37:36.15654376 +0000 UTC m=+1115.083181340"
Oct 30 23:37:37.140575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:37.140531    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc" event={"ID":"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9","Type":"ContainerStarted","Data":"ca8c5dd1ee46abf5703e74bfdce80958f3f414047e2ab1ceb29370e64f94c878"}
Oct 30 23:37:37.141074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:37.140655    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:37.141074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:37.140712    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 30 23:37:37.161883 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:37.161835    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc" podStartSLOduration=2.161306979 podStartE2EDuration="5.161820283s" totalImagesPullingTime="3.000513304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:37.160392345 +0000 UTC m=+1116.087029921" watchObservedRunningTime="2026-10-30 23:37:37.161820283 +0000 UTC m=+1116.088457845"
Oct 30 23:37:38.999290 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:38.999237    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:37:39.739557 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.739522    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"]
Oct 30 23:37:39.744924 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.744894    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.753342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.753096    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"]
Oct 30 23:37:39.759321 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.759290    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-b567bb-serving-cert\""
Oct 30 23:37:39.818504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.818460    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.818706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.818574    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghspj\" (UniqueName: \"kubernetes.io/projected/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-kube-api-access-ghspj\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.818706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.818648    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-metrics-certs\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.818824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.818715    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.919739 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.919700    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.919942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.919754    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.919942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.919805    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghspj\" (UniqueName: \"kubernetes.io/projected/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-kube-api-access-ghspj\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.919942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.919851    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-metrics-certs\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.920106 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:37:39.919939    4555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-serving-cert: secret "maas-api-e2e-worker-w3-b567bb-serving-cert" not found
Oct 30 23:37:39.920106 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:37:39.920023    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-maas-api-tls podName:6f4a512a-da26-4641-93d7-b6ae6cc73ffd nodeName:}" failed. No retries permitted until 2026-10-30 23:37:40.420000854 +0000 UTC m=+1119.346638430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-maas-api-tls") pod "maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc" (UID: "6f4a512a-da26-4641-93d7-b6ae6cc73ffd") : secret "maas-api-e2e-worker-w3-b567bb-serving-cert" not found
Oct 30 23:37:39.920491 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.920474    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.922412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.922388    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-metrics-certs\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:39.947074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:39.947042    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghspj\" (UniqueName: \"kubernetes.io/projected/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-kube-api-access-ghspj\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:40.424447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:40.424403    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:40.427061 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:40.427035    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:40.674605 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:40.674529    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:40.955031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:40.955006    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"]
Oct 30 23:37:40.957132 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:37:40.957093    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4a512a_da26_4641_93d7_b6ae6cc73ffd.slice/crio-2df77f13ac5066d449095ae89ae97ac456aea9eeccd9ca54cdfa32dde2c82118 WatchSource:0}: Error finding container 2df77f13ac5066d449095ae89ae97ac456aea9eeccd9ca54cdfa32dde2c82118: Status 404 returned error can't find the container with id 2df77f13ac5066d449095ae89ae97ac456aea9eeccd9ca54cdfa32dde2c82118
Oct 30 23:37:41.165873 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:41.165832    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc" event={"ID":"6f4a512a-da26-4641-93d7-b6ae6cc73ffd","Type":"ContainerStarted","Data":"2df77f13ac5066d449095ae89ae97ac456aea9eeccd9ca54cdfa32dde2c82118"}
Oct 30 23:37:42.172694 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:42.172643    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc" event={"ID":"6f4a512a-da26-4641-93d7-b6ae6cc73ffd","Type":"ContainerStarted","Data":"92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b"}
Oct 30 23:37:42.173112 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:42.172720    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:42.199691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:42.199641    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc" podStartSLOduration=2.668690004 podStartE2EDuration="3.199624691s" totalImagesPullingTime="530.934687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:42.197416645 +0000 UTC m=+1121.124054231" watchObservedRunningTime="2026-10-30 23:37:42.199624691 +0000 UTC m=+1121.126262273"
Oct 30 23:37:43.152231 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:43.152203    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:37:46.137019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.136990    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:37:46.521486 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.521454    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"]
Oct 30 23:37:46.526066 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.526045    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.535123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.535098    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"openshift-service-ca.crt\""
Oct 30 23:37:46.536047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.536022    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"kube-root-ca.crt\""
Oct 30 23:37:46.536218 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.536107    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"default-dockercfg-ftrjj\""
Oct 30 23:37:46.536218 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.536195    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dd3a15\"/\"simulator-w5-dd3a15-kserve-self-signed-certs\""
Oct 30 23:37:46.539618 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.539594    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"]
Oct 30 23:37:46.586723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.586678    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t8g9\" (UniqueName: \"kubernetes.io/projected/36867ec7-b651-4774-b248-c3f6b6696c69-kube-api-access-8t8g9\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.586901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.586746    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-home\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.587010 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.586993    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-dshm\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.587125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.587102    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-tmp-dir\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.587191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.587142    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-model-cache\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.587191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.587167    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36867ec7-b651-4774-b248-c3f6b6696c69-tls-certs\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.688615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.688573    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-tmp-dir\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.688825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.688623    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-model-cache\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.688825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.688649    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36867ec7-b651-4774-b248-c3f6b6696c69-tls-certs\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.688825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.688752    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t8g9\" (UniqueName: \"kubernetes.io/projected/36867ec7-b651-4774-b248-c3f6b6696c69-kube-api-access-8t8g9\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.688825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.688796    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-home\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.689061 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.688828    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-dshm\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.689061 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.688996    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-tmp-dir\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.689158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.689056    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-model-cache\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.689270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.689257    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-home\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.691303 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.691245    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-dshm\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.692034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.692013    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36867ec7-b651-4774-b248-c3f6b6696c69-tls-certs\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.701699 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.701676    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t8g9\" (UniqueName: \"kubernetes.io/projected/36867ec7-b651-4774-b248-c3f6b6696c69-kube-api-access-8t8g9\") pod \"simulator-w5-dd3a15-kserve-b84f58498-c7tnj\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") " pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:46.840820 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:46.840732    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:47.139095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:47.139069    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"]
Oct 30 23:37:47.140803 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:37:47.140759    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36867ec7_b651_4774_b248_c3f6b6696c69.slice/crio-e9c37bd4fbbe45b04d979a8f62cb2a70f506a8eaef60d073c5b3d7f977c7e3cf WatchSource:0}: Error finding container e9c37bd4fbbe45b04d979a8f62cb2a70f506a8eaef60d073c5b3d7f977c7e3cf: Status 404 returned error can't find the container with id e9c37bd4fbbe45b04d979a8f62cb2a70f506a8eaef60d073c5b3d7f977c7e3cf
Oct 30 23:37:47.143852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:47.143821    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 30 23:37:47.207052 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:47.207019    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" event={"ID":"36867ec7-b651-4774-b248-c3f6b6696c69","Type":"ContainerStarted","Data":"e9c37bd4fbbe45b04d979a8f62cb2a70f506a8eaef60d073c5b3d7f977c7e3cf"}
Oct 30 23:37:47.448731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:47.448636    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:37:48.150005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:48.149967    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 30 23:37:48.185806 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:48.185774    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 30 23:37:49.225860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:49.225818    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" event={"ID":"36867ec7-b651-4774-b248-c3f6b6696c69","Type":"ContainerStarted","Data":"7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be"}
Oct 30 23:37:49.246292 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:49.246214    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podStartSLOduration=1.966206372 podStartE2EDuration="3.246199648s" totalImagesPullingTime="1.279993276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:49.245519424 +0000 UTC m=+1128.172157004" watchObservedRunningTime="2026-10-30 23:37:49.246199648 +0000 UTC m=+1128.172837226"
Oct 30 23:37:49.841180 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:49.841145    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:53.990327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:53.990284    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:37:56.841454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:56.841406    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:56.860841 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:56.860814    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:57.287182 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:37:57.287142    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:37:57.448141 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:37:57.448104    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:38:00.356647 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.356610    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"]
Oct 30 23:38:00.360850 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.360822    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.372414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.372054    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"openshift-service-ca.crt\""
Oct 30 23:38:00.373779 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.373754    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"kube-root-ca.crt\""
Oct 30 23:38:00.376231 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.375675    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"default-dockercfg-l9wjz\""
Oct 30 23:38:00.376231 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.375829    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e15304\"/\"simulator-w6-e15304-kserve-self-signed-certs\""
Oct 30 23:38:00.377549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.377529    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"]
Oct 30 23:38:00.444197 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.444160    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-home\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.444346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.444212    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkvph\" (UniqueName: \"kubernetes.io/projected/5a9c4106-2c76-455b-9203-64bae7018a04-kube-api-access-lkvph\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.444417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.444351    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-dshm\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.444519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.444497    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-model-cache\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.444599 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.444586    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-tmp-dir\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.444663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.444649    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9c4106-2c76-455b-9203-64bae7018a04-tls-certs\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.545735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.545704    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-home\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.545920 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.545743    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkvph\" (UniqueName: \"kubernetes.io/projected/5a9c4106-2c76-455b-9203-64bae7018a04-kube-api-access-lkvph\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.545920 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.545797    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-dshm\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.545920 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.545823    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-model-cache\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.545920 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.545859    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-tmp-dir\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.545920 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.545905    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9c4106-2c76-455b-9203-64bae7018a04-tls-certs\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.546276 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.546249    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-home\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.546336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.546276    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-tmp-dir\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.546336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.546319    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-model-cache\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.548221 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.548194    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-dshm\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.548674 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.548653    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9c4106-2c76-455b-9203-64bae7018a04-tls-certs\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.558204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.558183    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkvph\" (UniqueName: \"kubernetes.io/projected/5a9c4106-2c76-455b-9203-64bae7018a04-kube-api-access-lkvph\") pod \"simulator-w6-e15304-kserve-74f6db8755-68g4g\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") " pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.680202 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.680117    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:00.971690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:00.971656    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"]
Oct 30 23:38:00.974220 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:38:00.974185    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a9c4106_2c76_455b_9203_64bae7018a04.slice/crio-d0fb593bdd44adc01e98b79a131245656793e4beadccf69ebd09eb3084284004 WatchSource:0}: Error finding container d0fb593bdd44adc01e98b79a131245656793e4beadccf69ebd09eb3084284004: Status 404 returned error can't find the container with id d0fb593bdd44adc01e98b79a131245656793e4beadccf69ebd09eb3084284004
Oct 30 23:38:01.307471 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:01.307416    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" event={"ID":"5a9c4106-2c76-455b-9203-64bae7018a04","Type":"ContainerStarted","Data":"d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782"}
Oct 30 23:38:01.307471 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:01.307466    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" event={"ID":"5a9c4106-2c76-455b-9203-64bae7018a04","Type":"ContainerStarted","Data":"d0fb593bdd44adc01e98b79a131245656793e4beadccf69ebd09eb3084284004"}
Oct 30 23:38:01.348131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:01.348063    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podStartSLOduration=1.3480424069999999 podStartE2EDuration="1.348042407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:01.346154291 +0000 UTC m=+1140.272791874" watchObservedRunningTime="2026-10-30 23:38:01.348042407 +0000 UTC m=+1140.274679989"
Oct 30 23:38:01.681181 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:01.681085    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:03.385217 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:03.385170    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:38:10.680502 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:10.680465    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:10.699317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:10.699287    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:11.380144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.380115    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:38:11.514866 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.514832    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"]
Oct 30 23:38:11.521982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.521951    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.529341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.529312    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"]
Oct 30 23:38:11.529675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.529651    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"simulator-w4-843d8c-kserve-self-signed-certs\""
Oct 30 23:38:11.529818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.529675    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"kube-root-ca.crt\""
Oct 30 23:38:11.529818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.529800    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"openshift-service-ca.crt\""
Oct 30 23:38:11.529917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.529815    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843d8c\"/\"default-dockercfg-75htb\""
Oct 30 23:38:11.573496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.573459    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4sjw\" (UniqueName: \"kubernetes.io/projected/56c6ddb0-5235-4458-a035-b94f0588a3bf-kube-api-access-p4sjw\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.573669 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.573508    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-dshm\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.573712 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.573669    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-tmp-dir\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.573847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.573821    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c6ddb0-5235-4458-a035-b94f0588a3bf-tls-certs\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.574001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.573865    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-model-cache\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.574001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.573989    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-home\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.675214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.675111    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c6ddb0-5235-4458-a035-b94f0588a3bf-tls-certs\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.675214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.675171    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-model-cache\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.675492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.675222    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-home\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.675492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.675255    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4sjw\" (UniqueName: \"kubernetes.io/projected/56c6ddb0-5235-4458-a035-b94f0588a3bf-kube-api-access-p4sjw\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.675492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.675289    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-dshm\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.675492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.675326    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-tmp-dir\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.675748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.675719    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-model-cache\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.675814 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.675756    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-tmp-dir\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.676144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.676120    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-home\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.678433 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.678405    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c6ddb0-5235-4458-a035-b94f0588a3bf-tls-certs\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.678541 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.678408    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-dshm\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.688381 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.688342    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4sjw\" (UniqueName: \"kubernetes.io/projected/56c6ddb0-5235-4458-a035-b94f0588a3bf-kube-api-access-p4sjw\") pod \"simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") " pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:11.835656 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:11.835617    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:12.126573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:12.126550    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"]
Oct 30 23:38:12.128890 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:38:12.128861    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56c6ddb0_5235_4458_a035_b94f0588a3bf.slice/crio-7871f9fc86d72cd4b591686cd14d55878e0fa7a63a2cb17ed97a70781f5565df WatchSource:0}: Error finding container 7871f9fc86d72cd4b591686cd14d55878e0fa7a63a2cb17ed97a70781f5565df: Status 404 returned error can't find the container with id 7871f9fc86d72cd4b591686cd14d55878e0fa7a63a2cb17ed97a70781f5565df
Oct 30 23:38:12.375475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:12.375440    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" event={"ID":"56c6ddb0-5235-4458-a035-b94f0588a3bf","Type":"ContainerStarted","Data":"fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a"}
Oct 30 23:38:12.375706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:12.375483    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" event={"ID":"56c6ddb0-5235-4458-a035-b94f0588a3bf","Type":"ContainerStarted","Data":"7871f9fc86d72cd4b591686cd14d55878e0fa7a63a2cb17ed97a70781f5565df"}
Oct 30 23:38:12.412667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:12.412603    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podStartSLOduration=1.4125872259999999 podStartE2EDuration="1.412587226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:12.411905345 +0000 UTC m=+1151.338542927" watchObservedRunningTime="2026-10-30 23:38:12.412587226 +0000 UTC m=+1151.339224807"
Oct 30 23:38:12.836462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:12.836429    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:15.757558 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:15.757522    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:38:21.836750 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:21.836710    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:21.850032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:21.850003    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:22.289415 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.287352    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"]
Oct 30 23:38:22.291941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.291917    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.299636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.299608    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"default-dockercfg-mdbds\""
Oct 30 23:38:22.311888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.301766    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"premium-w1-611ef3-kserve-self-signed-certs\""
Oct 30 23:38:22.311888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.310459    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"kube-root-ca.crt\""
Oct 30 23:38:22.311888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.310549    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"openshift-service-ca.crt\""
Oct 30 23:38:22.317076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.317051    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"]
Oct 30 23:38:22.401392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.401334    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/050aff6e-81f1-46f9-a18d-78e093d01888-tls-certs\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.401392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.401399    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-home\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.401630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.401515    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-tmp-dir\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.401630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.401589    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-model-cache\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.401630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.401627    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhc5\" (UniqueName: \"kubernetes.io/projected/050aff6e-81f1-46f9-a18d-78e093d01888-kube-api-access-7bhc5\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.401793 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.401731    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-dshm\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.464030 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.463985    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:38:22.502548 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.502511    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-tmp-dir\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.502744 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.502574    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-model-cache\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.502744 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.502606    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhc5\" (UniqueName: \"kubernetes.io/projected/050aff6e-81f1-46f9-a18d-78e093d01888-kube-api-access-7bhc5\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.503282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.503255    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-tmp-dir\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.503459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.503315    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-model-cache\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.503559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.503406    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-dshm\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.503753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.503736    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/050aff6e-81f1-46f9-a18d-78e093d01888-tls-certs\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.503872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.503860    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-home\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.504259 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.504236    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-home\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.507627 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.507584    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/050aff6e-81f1-46f9-a18d-78e093d01888-tls-certs\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.511909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.511888    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-dshm\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.516043 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.515993    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhc5\" (UniqueName: \"kubernetes.io/projected/050aff6e-81f1-46f9-a18d-78e093d01888-kube-api-access-7bhc5\") pod \"premium-w1-611ef3-kserve-5f7df9b957-hlk9w\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") " pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.614849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.614766    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:22.898736 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:22.898711    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"]
Oct 30 23:38:22.900351 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:38:22.900325    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod050aff6e_81f1_46f9_a18d_78e093d01888.slice/crio-f73551805f03af43d3c921d192e78e8bee22036f0449775161a43d4f77ae2886 WatchSource:0}: Error finding container f73551805f03af43d3c921d192e78e8bee22036f0449775161a43d4f77ae2886: Status 404 returned error can't find the container with id f73551805f03af43d3c921d192e78e8bee22036f0449775161a43d4f77ae2886
Oct 30 23:38:23.456954 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:23.456909    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" event={"ID":"050aff6e-81f1-46f9-a18d-78e093d01888","Type":"ContainerStarted","Data":"603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264"}
Oct 30 23:38:23.457169 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:23.456964    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" event={"ID":"050aff6e-81f1-46f9-a18d-78e093d01888","Type":"ContainerStarted","Data":"f73551805f03af43d3c921d192e78e8bee22036f0449775161a43d4f77ae2886"}
Oct 30 23:38:23.479380 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:23.479320    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podStartSLOduration=1.479306467 podStartE2EDuration="1.479306467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:23.477191872 +0000 UTC m=+1162.403829454" watchObservedRunningTime="2026-10-30 23:38:23.479306467 +0000 UTC m=+1162.405944104"
Oct 30 23:38:23.614874 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:23.614850    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:26.145279 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:26.145241    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:38:27.445908 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:38:27.445871    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:38:32.615223 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:32.615171    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:32.628187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:32.628156    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:33.036185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.036152    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"]
Oct 30 23:38:33.040922 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.040894    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.049363 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.049144    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"kube-root-ca.crt\""
Oct 30 23:38:33.049363 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.049186    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"default-dockercfg-4fxkj\""
Oct 30 23:38:33.049363 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.049232    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"openshift-service-ca.crt\""
Oct 30 23:38:33.049363 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.049265    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"premium-w2-9ef03a-kserve-self-signed-certs\""
Oct 30 23:38:33.051596 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.051569    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"]
Oct 30 23:38:33.113819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.113782    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddac194a-9d84-41d6-b40b-e7aa74945352-tls-certs\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.114014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.113840    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-dshm\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.114014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.113976    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-home\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.114121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.114016    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-tmp-dir\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.114121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.114079    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zccrc\" (UniqueName: \"kubernetes.io/projected/ddac194a-9d84-41d6-b40b-e7aa74945352-kube-api-access-zccrc\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.114227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.114128    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-model-cache\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.214677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.214636    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-tmp-dir\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.214677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.214681    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zccrc\" (UniqueName: \"kubernetes.io/projected/ddac194a-9d84-41d6-b40b-e7aa74945352-kube-api-access-zccrc\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.214945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.214720    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-model-cache\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.214945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.214800    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddac194a-9d84-41d6-b40b-e7aa74945352-tls-certs\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.214945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.214835    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-dshm\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.214945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.214900    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-home\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.215553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.215518    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-model-cache\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.215802 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.215784    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-home\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.216753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.216726    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-tmp-dir\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.229451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.220737    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddac194a-9d84-41d6-b40b-e7aa74945352-tls-certs\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.229451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.220921    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-dshm\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.230216 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.230185    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zccrc\" (UniqueName: \"kubernetes.io/projected/ddac194a-9d84-41d6-b40b-e7aa74945352-kube-api-access-zccrc\") pod \"premium-w2-9ef03a-kserve-564b59bf56-8jdr8\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") " pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.356342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.356259    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:33.530433 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.530407    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:38:33.644888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:33.644863    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"]
Oct 30 23:38:33.646602 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:38:33.646572    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddac194a_9d84_41d6_b40b_e7aa74945352.slice/crio-3a2ad81811396fa6f9a758e46d252a0dbb0dddd754f4e3af714b371dac3f346a WatchSource:0}: Error finding container 3a2ad81811396fa6f9a758e46d252a0dbb0dddd754f4e3af714b371dac3f346a: Status 404 returned error can't find the container with id 3a2ad81811396fa6f9a758e46d252a0dbb0dddd754f4e3af714b371dac3f346a
Oct 30 23:38:34.527342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:34.527307    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" event={"ID":"ddac194a-9d84-41d6-b40b-e7aa74945352","Type":"ContainerStarted","Data":"1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1"}
Oct 30 23:38:34.527342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:34.527348    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" event={"ID":"ddac194a-9d84-41d6-b40b-e7aa74945352","Type":"ContainerStarted","Data":"3a2ad81811396fa6f9a758e46d252a0dbb0dddd754f4e3af714b371dac3f346a"}
Oct 30 23:38:34.547533 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:34.547456    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podStartSLOduration=1.547442405 podStartE2EDuration="1.547442405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:34.54615325 +0000 UTC m=+1173.472790831" watchObservedRunningTime="2026-10-30 23:38:34.547442405 +0000 UTC m=+1173.474079985"
Oct 30 23:38:35.356627 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:35.356593    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:40.617921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.617879    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"]
Oct 30 23:38:40.621859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.621843    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.629006 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.628957    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dd3a15-openshift-default-dockercfg-72cjz\""
Oct 30 23:38:40.637066 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.637039    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"]
Oct 30 23:38:40.677014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.676969    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3943f43f-9105-48e1-b60b-2ca22b65aa80-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.677202 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.677032    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.677202 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.677162    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.677315 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.677204    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.677426 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.677405    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.677500 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.677445    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldnfl\" (UniqueName: \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-kube-api-access-ldnfl\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.677500 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.677486    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.677589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.677572    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.677643 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.677627    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.778801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.778764    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3943f43f-9105-48e1-b60b-2ca22b65aa80-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.778819    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.778853    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.778876    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.778971    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.778998    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldnfl\" (UniqueName: \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-kube-api-access-ldnfl\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.779037    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.779080    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.779162    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779525 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.779358    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779525 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.779404    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779642 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.779596    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.779689 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.779647    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3943f43f-9105-48e1-b60b-2ca22b65aa80-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.781703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.780062    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.793083 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.793051    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.794833 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.794709    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.795160 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.795137    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.795702 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.795676    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldnfl\" (UniqueName: \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-kube-api-access-ldnfl\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:40.939504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:40.939409    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:41.235329 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:41.235298    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"]
Oct 30 23:38:41.236932 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:38:41.236901    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3943f43f_9105_48e1_b60b_2ca22b65aa80.slice/crio-00318b5b48dfb78d8cd467c60de21002497d65879548d215c5ff03a339b4e0a7 WatchSource:0}: Error finding container 00318b5b48dfb78d8cd467c60de21002497d65879548d215c5ff03a339b4e0a7: Status 404 returned error can't find the container with id 00318b5b48dfb78d8cd467c60de21002497d65879548d215c5ff03a339b4e0a7
Oct 30 23:38:41.239382 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:41.239333    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:38:41.239466 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:41.239421    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:38:41.239466 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:41.239451    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:38:41.573467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:41.573432    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" event={"ID":"3943f43f-9105-48e1-b60b-2ca22b65aa80","Type":"ContainerStarted","Data":"8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2"}
Oct 30 23:38:41.573778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:41.573747    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" event={"ID":"3943f43f-9105-48e1-b60b-2ca22b65aa80","Type":"ContainerStarted","Data":"00318b5b48dfb78d8cd467c60de21002497d65879548d215c5ff03a339b4e0a7"}
Oct 30 23:38:41.602835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:41.602770    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podStartSLOduration=1.602750807 podStartE2EDuration="1.602750807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:41.597030469 +0000 UTC m=+1180.523668051" watchObservedRunningTime="2026-10-30 23:38:41.602750807 +0000 UTC m=+1180.529388389"
Oct 30 23:38:41.941892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:41.941803    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:42.941388 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:42.941340    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:38:42.941581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:42.941421    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 23:38:43.357242 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:43.357199    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:43.410993 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:43.410961    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:43.617744 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:43.617664    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:38:43.940709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:43.940628    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:38:43.940709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:43.940680    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 23:38:44.939855 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:44.939820    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:38:44.940263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:44.939873    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 23:38:44.993889 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:44.993004    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:38:45.940572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:45.940529    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:38:45.940995 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:45.940594    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 23:38:46.940535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:46.940501    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:38:46.940733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:46.940557    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 23:38:47.940252 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:47.940209    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:38:47.940470 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:47.940285    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 23:38:49.262902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:49.262872    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:49.263448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:49.263396    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:49.265166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:49.265141    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 30 23:38:55.890067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.890029    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"]
Oct 30 23:38:55.894201 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.894182    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:55.907061 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.907019    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"]
Oct 30 23:38:55.939926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.939896    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3dcca-openshift-default-dockercfg-mtkk6\""
Oct 30 23:38:55.974760 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.974725    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:55.974940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.974779    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:55.974940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.974798    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:55.974940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.974899    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:55.974940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.974924    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:55.975073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.974942    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:55.975109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.975087    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde9335c-c895-4cd1-a70b-5823203f84b0-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:55.975207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.975186    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:55.975243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:55.975230    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9zl\" (UniqueName: \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-kube-api-access-wq9zl\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.076595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.076550    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.076595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.076599    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.076854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.076628    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.076854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.076656    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.076854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.076678    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.076854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.076745    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde9335c-c895-4cd1-a70b-5823203f84b0-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.076854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.076798    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.076854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.076849    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9zl\" (UniqueName: \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-kube-api-access-wq9zl\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.077157 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.076925    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.077280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.077248    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.077355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.077333    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.077422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.077406    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.077826 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.077800    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde9335c-c895-4cd1-a70b-5823203f84b0-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.077944 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.077925    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.079500 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.079480    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.079646 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.079628    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.089193 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.089171    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.089359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.089336    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9zl\" (UniqueName: \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-kube-api-access-wq9zl\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.209906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.209814    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:56.497317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.497287    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"]
Oct 30 23:38:56.499183 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:38:56.499155    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde9335c_c895_4cd1_a70b_5823203f84b0.slice/crio-f8dce0caeb5c7f9df1801772c057df2928e20f4c3b446384956fa179d82a2c88 WatchSource:0}: Error finding container f8dce0caeb5c7f9df1801772c057df2928e20f4c3b446384956fa179d82a2c88: Status 404 returned error can't find the container with id f8dce0caeb5c7f9df1801772c057df2928e20f4c3b446384956fa179d82a2c88
Oct 30 23:38:56.501757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.501722    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:38:56.501848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.501798    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:38:56.501848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.501830    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:38:56.689261 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.689224    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" event={"ID":"cde9335c-c895-4cd1-a70b-5823203f84b0","Type":"ContainerStarted","Data":"7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9"}
Oct 30 23:38:56.689458 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.689286    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" event={"ID":"cde9335c-c895-4cd1-a70b-5823203f84b0","Type":"ContainerStarted","Data":"f8dce0caeb5c7f9df1801772c057df2928e20f4c3b446384956fa179d82a2c88"}
Oct 30 23:38:56.714964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:56.714904    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" podStartSLOduration=1.714886117 podStartE2EDuration="1.714886117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:56.714259061 +0000 UTC m=+1195.640896643" watchObservedRunningTime="2026-10-30 23:38:56.714886117 +0000 UTC m=+1195.641523697"
Oct 30 23:38:57.210030 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:57.209997    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:38:57.211520 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:57.211494    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:38:57.211644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:57.211544    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:38:57.446435 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:38:57.446398    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:38:58.210400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:58.210339    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:38:58.210851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:58.210408    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:38:59.211084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:59.211046    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:38:59.211492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:38:59.211124    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:39:00.211120 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:00.211078    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:39:00.211566 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:00.211141    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:39:01.210647 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:01.210613    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:39:01.210873 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:01.210684    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:39:02.210951 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:02.210914    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:39:02.211408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:02.210970    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:39:03.211274 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:03.211238    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 23:39:03.211832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:03.211316    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 23:39:04.500796 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:04.500765    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:39:04.501227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:04.501079    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:39:04.501907 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:04.501891    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 30 23:39:11.082866 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.082831    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"]
Oct 30 23:39:11.087681 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.087665    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.093246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.093222    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b567bb-openshift-default-dockercfg-wvjgm\""
Oct 30 23:39:11.101718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.101692    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"]
Oct 30 23:39:11.241036 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.240945    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.241263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.241076    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.241263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.241138    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.241263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.241163    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.241263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.241198    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.241263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.241223    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.241503 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.241265    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.241503 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.241389    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.241503 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.241414    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pljd2\" (UniqueName: \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-kube-api-access-pljd2\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.342535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.342270    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.342535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.342316    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pljd2\" (UniqueName: \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-kube-api-access-pljd2\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.342535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.342365    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.342535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.342430    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.342535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.342490    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.342535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.342517    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.342982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.342555    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.342982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.342586    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.342982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.342630    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.342982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.342877    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.343197 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.343091    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.343254 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.343191    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.343254 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.343191    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.343584 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.343558    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.345172 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.345150    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.345642 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.345619    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.355727 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.355694    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.355818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.355706    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pljd2\" (UniqueName: \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-kube-api-access-pljd2\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.401605 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.401568    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:11.693642 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.693615    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"]
Oct 30 23:39:11.694746 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:11.694710    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73713bb_0f2b_4f38_8d81_bb0b437348d1.slice/crio-b45b31d30a69561073e89f5e3a9946bcc94d9d67b1d6d529ccd3ead242007f2b WatchSource:0}: Error finding container b45b31d30a69561073e89f5e3a9946bcc94d9d67b1d6d529ccd3ead242007f2b: Status 404 returned error can't find the container with id b45b31d30a69561073e89f5e3a9946bcc94d9d67b1d6d529ccd3ead242007f2b
Oct 30 23:39:11.697136 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.697106    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:39:11.697215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.697181    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:39:11.697215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.697210    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:39:11.800433 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.800393    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" event={"ID":"c73713bb-0f2b-4f38-8d81-bb0b437348d1","Type":"ContainerStarted","Data":"46a46e95cbf0ce814741316fba13c7214a03bd83759646d5016d42fead150e2f"}
Oct 30 23:39:11.800611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.800461    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" event={"ID":"c73713bb-0f2b-4f38-8d81-bb0b437348d1","Type":"ContainerStarted","Data":"b45b31d30a69561073e89f5e3a9946bcc94d9d67b1d6d529ccd3ead242007f2b"}
Oct 30 23:39:11.825355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:11.825292    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" podStartSLOduration=0.825273413 podStartE2EDuration="825.273413ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:11.823028157 +0000 UTC m=+1210.749665739" watchObservedRunningTime="2026-10-30 23:39:11.825273413 +0000 UTC m=+1210.751910992"
Oct 30 23:39:12.401912 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:12.401877    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:12.403629 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:12.403604    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:39:12.403744 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:12.403651    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:39:13.401887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:13.401850    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:39:13.402065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:13.401900    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:39:14.401961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:14.401928    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:39:14.402141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:14.401979    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:39:15.401975 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:15.401938    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:39:15.402189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:15.401989    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:39:16.281391 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.281340    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"]
Oct 30 23:39:16.306695 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.306660    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.307013 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.306986    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"]
Oct 30 23:39:16.311457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.311429    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd231773-fef2-4945-bcb9-e09329508200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4108876-g4j82\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.311582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.311496    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sss45\" (UniqueName: \"kubernetes.io/projected/fd231773-fef2-4945-bcb9-e09329508200-kube-api-access-sss45\") pod \"maas-api-revoke-sub-e4108876-g4j82\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.311645 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.311613    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd231773-fef2-4945-bcb9-e09329508200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4108876-g4j82\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.311781 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.311764    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:39:16.311934 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.311917    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:39:16.402289 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.402252    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:39:16.402797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.402330    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:39:16.412437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.412404    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd231773-fef2-4945-bcb9-e09329508200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4108876-g4j82\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.412647 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.412453    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sss45\" (UniqueName: \"kubernetes.io/projected/fd231773-fef2-4945-bcb9-e09329508200-kube-api-access-sss45\") pod \"maas-api-revoke-sub-e4108876-g4j82\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.412647 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.412512    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd231773-fef2-4945-bcb9-e09329508200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4108876-g4j82\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.413397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.413355    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd231773-fef2-4945-bcb9-e09329508200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4108876-g4j82\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.415095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.415073    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd231773-fef2-4945-bcb9-e09329508200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4108876-g4j82\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.425984 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.425957    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sss45\" (UniqueName: \"kubernetes.io/projected/fd231773-fef2-4945-bcb9-e09329508200-kube-api-access-sss45\") pod \"maas-api-revoke-sub-e4108876-g4j82\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.623078 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.622987    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:16.949961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.949932    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"]
Oct 30 23:39:16.953611 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:16.953587    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd231773_fef2_4945_bcb9_e09329508200.slice/crio-6d7741ac1b17cde437b261613d4585574a1aa944961f3981b55a8e86038f6339 WatchSource:0}: Error finding container 6d7741ac1b17cde437b261613d4585574a1aa944961f3981b55a8e86038f6339: Status 404 returned error can't find the container with id 6d7741ac1b17cde437b261613d4585574a1aa944961f3981b55a8e86038f6339
Oct 30 23:39:16.955543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:16.955523    4555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:39:17.403825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:17.403491    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 23:39:17.403825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:17.403540    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 23:39:17.844648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:17.844612    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82" event={"ID":"fd231773-fef2-4945-bcb9-e09329508200","Type":"ContainerStarted","Data":"6d7741ac1b17cde437b261613d4585574a1aa944961f3981b55a8e86038f6339"}
Oct 30 23:39:18.243847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:18.243767    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:39:18.663039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:18.662951    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:18.664697 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:18.664324    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:18.665246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:18.665187    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 30 23:39:20.868255 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:20.868217    4555 generic.go:356] "Generic (PLEG): container finished" podID="fd231773-fef2-4945-bcb9-e09329508200" containerID="f39e90beb935d35e872998360c4bf57bf33d9d4c26761a6fef6efdb98882f62b" exitCode=0
Oct 30 23:39:20.868679 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:20.868313    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82" event={"ID":"fd231773-fef2-4945-bcb9-e09329508200","Type":"ContainerDied","Data":"f39e90beb935d35e872998360c4bf57bf33d9d4c26761a6fef6efdb98882f62b"}
Oct 30 23:39:22.026090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.026056    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:22.083801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.083769    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd231773-fef2-4945-bcb9-e09329508200-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd231773-fef2-4945-bcb9-e09329508200-maas-api-xks-ca\") pod \"fd231773-fef2-4945-bcb9-e09329508200\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") "
Oct 30 23:39:22.083996 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.083828    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd231773-fef2-4945-bcb9-e09329508200-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd231773-fef2-4945-bcb9-e09329508200-maas-api-openshift-ca\") pod \"fd231773-fef2-4945-bcb9-e09329508200\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") "
Oct 30 23:39:22.083996 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.083905    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd231773-fef2-4945-bcb9-e09329508200-kube-api-access-sss45\" (UniqueName: \"kubernetes.io/projected/fd231773-fef2-4945-bcb9-e09329508200-kube-api-access-sss45\") pod \"fd231773-fef2-4945-bcb9-e09329508200\" (UID: \"fd231773-fef2-4945-bcb9-e09329508200\") "
Oct 30 23:39:22.084209 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.084187    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd231773-fef2-4945-bcb9-e09329508200-maas-api-openshift-ca" pod "fd231773-fef2-4945-bcb9-e09329508200" (UID: "fd231773-fef2-4945-bcb9-e09329508200"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:39:22.084339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.084322    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd231773-fef2-4945-bcb9-e09329508200-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:22.086282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.086253    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd231773-fef2-4945-bcb9-e09329508200-kube-api-access-sss45" pod "fd231773-fef2-4945-bcb9-e09329508200" (UID: "fd231773-fef2-4945-bcb9-e09329508200"). InnerVolumeSpecName "kube-api-access-sss45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:39:22.086462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.086389    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd231773-fef2-4945-bcb9-e09329508200-maas-api-xks-ca" pod "fd231773-fef2-4945-bcb9-e09329508200" (UID: "fd231773-fef2-4945-bcb9-e09329508200"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:39:22.185495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.185401    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd231773-fef2-4945-bcb9-e09329508200-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:22.185495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.185429    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sss45\" (UniqueName: \"kubernetes.io/projected/fd231773-fef2-4945-bcb9-e09329508200-kube-api-access-sss45\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:22.883073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.883040    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"
Oct 30 23:39:22.883225 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.883039    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82" event={"ID":"fd231773-fef2-4945-bcb9-e09329508200","Type":"ContainerDied","Data":"6d7741ac1b17cde437b261613d4585574a1aa944961f3981b55a8e86038f6339"}
Oct 30 23:39:22.883225 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:22.883164    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d7741ac1b17cde437b261613d4585574a1aa944961f3981b55a8e86038f6339"
Oct 30 23:39:27.446775 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:39:27.446732    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:39:31.267487 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.267453    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"]
Oct 30 23:39:31.267942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.267884    4555 cpu_manager.go:414] "containerMap: removing container" podUID="fd231773-fef2-4945-bcb9-e09329508200" containerName="revoke-keys"
Oct 30 23:39:31.267942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.267895    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd231773-fef2-4945-bcb9-e09329508200" containerName="revoke-keys"
Oct 30 23:39:31.268018 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.267960    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd231773-fef2-4945-bcb9-e09329508200" containerName="revoke-keys"
Oct 30 23:39:31.271563 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.271542    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.279915 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.279894    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:39:31.280579 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.280562    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:39:31.281464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.281432    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"]
Oct 30 23:39:31.383125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.383094    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh26w\" (UniqueName: \"kubernetes.io/projected/10ab930a-55b4-46ea-9497-40fa4a329602-kube-api-access-zh26w\") pod \"maas-api-revoke-sub-350338c9-gjkr6\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.383319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.383218    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-350338c9-gjkr6\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.383319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.383299    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-350338c9-gjkr6\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.484488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.484455    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh26w\" (UniqueName: \"kubernetes.io/projected/10ab930a-55b4-46ea-9497-40fa4a329602-kube-api-access-zh26w\") pod \"maas-api-revoke-sub-350338c9-gjkr6\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.484663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.484524    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-350338c9-gjkr6\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.484663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.484585    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-350338c9-gjkr6\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.485538 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.485507    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-350338c9-gjkr6\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.487144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.487115    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-350338c9-gjkr6\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.496853 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.496832    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh26w\" (UniqueName: \"kubernetes.io/projected/10ab930a-55b4-46ea-9497-40fa4a329602-kube-api-access-zh26w\") pod \"maas-api-revoke-sub-350338c9-gjkr6\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.588524 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.588493    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:31.870743 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.870663    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"]
Oct 30 23:39:31.873897 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:31.873865    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ab930a_55b4_46ea_9497_40fa4a329602.slice/crio-97d38cc9994574b47bfaa6acebcaec4e317915c2931a8c9aa6f7d36f9131ebac WatchSource:0}: Error finding container 97d38cc9994574b47bfaa6acebcaec4e317915c2931a8c9aa6f7d36f9131ebac: Status 404 returned error can't find the container with id 97d38cc9994574b47bfaa6acebcaec4e317915c2931a8c9aa6f7d36f9131ebac
Oct 30 23:39:31.947429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:31.947397    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6" event={"ID":"10ab930a-55b4-46ea-9497-40fa4a329602","Type":"ContainerStarted","Data":"97d38cc9994574b47bfaa6acebcaec4e317915c2931a8c9aa6f7d36f9131ebac"}
Oct 30 23:39:32.954667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:32.954570    4555 generic.go:356] "Generic (PLEG): container finished" podID="10ab930a-55b4-46ea-9497-40fa4a329602" containerID="535425922ea5f5935f1b9309539ad48a50dce480e481e88531c98ef9883cd6c0" exitCode=0
Oct 30 23:39:32.954667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:32.954652    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6" event={"ID":"10ab930a-55b4-46ea-9497-40fa4a329602","Type":"ContainerDied","Data":"535425922ea5f5935f1b9309539ad48a50dce480e481e88531c98ef9883cd6c0"}
Oct 30 23:39:33.056328 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.056297    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"]
Oct 30 23:39:33.060310 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.060293    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.069164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.069132    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"]
Oct 30 23:39:33.205565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.205435    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f64a45-jdhw6\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.205733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.205576    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f64a45-jdhw6\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.205733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.205720    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkpd\" (UniqueName: \"kubernetes.io/projected/2c052a30-2727-4348-9dbd-5e938012993c-kube-api-access-cvkpd\") pod \"maas-api-revoke-sub-58f64a45-jdhw6\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.306306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.306271    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkpd\" (UniqueName: \"kubernetes.io/projected/2c052a30-2727-4348-9dbd-5e938012993c-kube-api-access-cvkpd\") pod \"maas-api-revoke-sub-58f64a45-jdhw6\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.306502 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.306339    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f64a45-jdhw6\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.306502 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.306471    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f64a45-jdhw6\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.307161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.307136    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f64a45-jdhw6\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.308998 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.308965    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f64a45-jdhw6\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.319436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.319415    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkpd\" (UniqueName: \"kubernetes.io/projected/2c052a30-2727-4348-9dbd-5e938012993c-kube-api-access-cvkpd\") pod \"maas-api-revoke-sub-58f64a45-jdhw6\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.375106 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.375076    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:33.658475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.658436    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"]
Oct 30 23:39:33.660049 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:33.660026    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c052a30_2727_4348_9dbd_5e938012993c.slice/crio-13f4e146cd0f372438ca8b39efef4f2ee417c69ce309d75ae6af969ab96e196b WatchSource:0}: Error finding container 13f4e146cd0f372438ca8b39efef4f2ee417c69ce309d75ae6af969ab96e196b: Status 404 returned error can't find the container with id 13f4e146cd0f372438ca8b39efef4f2ee417c69ce309d75ae6af969ab96e196b
Oct 30 23:39:33.975195 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:33.975151    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6" event={"ID":"2c052a30-2727-4348-9dbd-5e938012993c","Type":"ContainerStarted","Data":"13f4e146cd0f372438ca8b39efef4f2ee417c69ce309d75ae6af969ab96e196b"}
Oct 30 23:39:34.131933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.131906    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:34.216734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.216699    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ab930a-55b4-46ea-9497-40fa4a329602-kube-api-access-zh26w\" (UniqueName: \"kubernetes.io/projected/10ab930a-55b4-46ea-9497-40fa4a329602-kube-api-access-zh26w\") pod \"10ab930a-55b4-46ea-9497-40fa4a329602\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") "
Oct 30 23:39:34.216902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.216828    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-openshift-ca\") pod \"10ab930a-55b4-46ea-9497-40fa4a329602\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") "
Oct 30 23:39:34.216953 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.216904    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-xks-ca\") pod \"10ab930a-55b4-46ea-9497-40fa4a329602\" (UID: \"10ab930a-55b4-46ea-9497-40fa4a329602\") "
Oct 30 23:39:34.217212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.217187    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-openshift-ca" pod "10ab930a-55b4-46ea-9497-40fa4a329602" (UID: "10ab930a-55b4-46ea-9497-40fa4a329602"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:39:34.219028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.219004    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-xks-ca" pod "10ab930a-55b4-46ea-9497-40fa4a329602" (UID: "10ab930a-55b4-46ea-9497-40fa4a329602"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:39:34.219117 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.219078    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ab930a-55b4-46ea-9497-40fa4a329602-kube-api-access-zh26w" pod "10ab930a-55b4-46ea-9497-40fa4a329602" (UID: "10ab930a-55b4-46ea-9497-40fa4a329602"). InnerVolumeSpecName "kube-api-access-zh26w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:39:34.317890 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.317864    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh26w\" (UniqueName: \"kubernetes.io/projected/10ab930a-55b4-46ea-9497-40fa4a329602-kube-api-access-zh26w\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:34.317890 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.317889    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:34.318091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.317899    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ab930a-55b4-46ea-9497-40fa4a329602-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:34.983080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.983043    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"
Oct 30 23:39:34.983080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.983072    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6" event={"ID":"10ab930a-55b4-46ea-9497-40fa4a329602","Type":"ContainerDied","Data":"97d38cc9994574b47bfaa6acebcaec4e317915c2931a8c9aa6f7d36f9131ebac"}
Oct 30 23:39:34.983648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.983123    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97d38cc9994574b47bfaa6acebcaec4e317915c2931a8c9aa6f7d36f9131ebac"
Oct 30 23:39:34.984968 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.984933    4555 generic.go:356] "Generic (PLEG): container finished" podID="2c052a30-2727-4348-9dbd-5e938012993c" containerID="fb702a8ff7660d802618be92731febb0aa444b1d47ed610d1e68556e94a895d4" exitCode=0
Oct 30 23:39:34.985120 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:34.985020    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6" event={"ID":"2c052a30-2727-4348-9dbd-5e938012993c","Type":"ContainerDied","Data":"fb702a8ff7660d802618be92731febb0aa444b1d47ed610d1e68556e94a895d4"}
Oct 30 23:39:36.136849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:36.136826    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:36.239426 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:36.239390    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-xks-ca\") pod \"2c052a30-2727-4348-9dbd-5e938012993c\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") "
Oct 30 23:39:36.239598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:36.239473    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-openshift-ca\") pod \"2c052a30-2727-4348-9dbd-5e938012993c\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") "
Oct 30 23:39:36.239598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:36.239575    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c052a30-2727-4348-9dbd-5e938012993c-kube-api-access-cvkpd\" (UniqueName: \"kubernetes.io/projected/2c052a30-2727-4348-9dbd-5e938012993c-kube-api-access-cvkpd\") pod \"2c052a30-2727-4348-9dbd-5e938012993c\" (UID: \"2c052a30-2727-4348-9dbd-5e938012993c\") "
Oct 30 23:39:36.239884 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:36.239856    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-openshift-ca" pod "2c052a30-2727-4348-9dbd-5e938012993c" (UID: "2c052a30-2727-4348-9dbd-5e938012993c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:39:36.240003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:36.239950    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:36.241819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:36.241762    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-xks-ca" pod "2c052a30-2727-4348-9dbd-5e938012993c" (UID: "2c052a30-2727-4348-9dbd-5e938012993c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:39:36.241902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:36.241805    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c052a30-2727-4348-9dbd-5e938012993c-kube-api-access-cvkpd" pod "2c052a30-2727-4348-9dbd-5e938012993c" (UID: "2c052a30-2727-4348-9dbd-5e938012993c"). InnerVolumeSpecName "kube-api-access-cvkpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:39:36.340615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:36.340575    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c052a30-2727-4348-9dbd-5e938012993c-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:36.340615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:36.340603    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvkpd\" (UniqueName: \"kubernetes.io/projected/2c052a30-2727-4348-9dbd-5e938012993c-kube-api-access-cvkpd\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:37.000897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:37.000870    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"
Oct 30 23:39:37.001070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:37.000900    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6" event={"ID":"2c052a30-2727-4348-9dbd-5e938012993c","Type":"ContainerDied","Data":"13f4e146cd0f372438ca8b39efef4f2ee417c69ce309d75ae6af969ab96e196b"}
Oct 30 23:39:37.001070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:37.000947    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13f4e146cd0f372438ca8b39efef4f2ee417c69ce309d75ae6af969ab96e196b"
Oct 30 23:39:40.439623 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.439588    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"]
Oct 30 23:39:40.440521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.440495    4555 cpu_manager.go:414] "containerMap: removing container" podUID="10ab930a-55b4-46ea-9497-40fa4a329602" containerName="revoke-keys"
Oct 30 23:39:40.440521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.440519    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ab930a-55b4-46ea-9497-40fa4a329602" containerName="revoke-keys"
Oct 30 23:39:40.440713 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.440531    4555 cpu_manager.go:414] "containerMap: removing container" podUID="2c052a30-2727-4348-9dbd-5e938012993c" containerName="revoke-keys"
Oct 30 23:39:40.440713 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.440540    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c052a30-2727-4348-9dbd-5e938012993c" containerName="revoke-keys"
Oct 30 23:39:40.440830 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.440718    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ab930a-55b4-46ea-9497-40fa4a329602" containerName="revoke-keys"
Oct 30 23:39:40.440830 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.440731    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c052a30-2727-4348-9dbd-5e938012993c" containerName="revoke-keys"
Oct 30 23:39:40.459715 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.459683    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.461457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.461081    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"]
Oct 30 23:39:40.469400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.469356    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:39:40.470384 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.470343    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:39:40.584067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.584032    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504a4e52-2rv9n\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.584268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.584219    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504a4e52-2rv9n\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.584333 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.584265    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gcl\" (UniqueName: \"kubernetes.io/projected/a4526b5d-ba8d-410e-8f1c-717ea858c74e-kube-api-access-42gcl\") pod \"maas-api-revoke-sub-504a4e52-2rv9n\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.685238 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.685206    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504a4e52-2rv9n\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.685436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.685246    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gcl\" (UniqueName: \"kubernetes.io/projected/a4526b5d-ba8d-410e-8f1c-717ea858c74e-kube-api-access-42gcl\") pod \"maas-api-revoke-sub-504a4e52-2rv9n\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.685436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.685286    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504a4e52-2rv9n\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.686069 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.685984    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504a4e52-2rv9n\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.689544 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.688404    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504a4e52-2rv9n\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.701646 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.701580    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gcl\" (UniqueName: \"kubernetes.io/projected/a4526b5d-ba8d-410e-8f1c-717ea858c74e-kube-api-access-42gcl\") pod \"maas-api-revoke-sub-504a4e52-2rv9n\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.778568 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.778537    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"]
Oct 30 23:39:40.781786 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.781762    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:40.783024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.783009    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.786914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.786891    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-611ef3-openshift-default-dockercfg-gcsfx\""
Oct 30 23:39:40.794951 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.794921    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"]
Oct 30 23:39:40.888051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.887346    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.888051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.887475    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.888051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.887529    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.888051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.887561    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.888051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.887610    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gtc5\" (UniqueName: \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-kube-api-access-8gtc5\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.888051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.887641    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.888051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.887678    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.888051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.887717    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.888051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.887767    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.988923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.988829    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.988923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.988888    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.988923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.988915    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989225 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.988933    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989225 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.988974    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gtc5\" (UniqueName: \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-kube-api-access-8gtc5\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989225 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.989001    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989225 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.989028    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989225 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.989062    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989225 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.989107    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989567 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.989444    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989707 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.989682    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.989803    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989904 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.989828    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.989991 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.989972    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.991663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.991632    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:40.991755 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:40.991633    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:41.001847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:41.001828    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:41.002095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:41.002078    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gtc5\" (UniqueName: \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-kube-api-access-8gtc5\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:41.076525 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:41.076491    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"]
Oct 30 23:39:41.080653 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:41.080625    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4526b5d_ba8d_410e_8f1c_717ea858c74e.slice/crio-ef9493c51ad465b720972aae2223b77bac43d786a99662db70b8c9eddba6513f WatchSource:0}: Error finding container ef9493c51ad465b720972aae2223b77bac43d786a99662db70b8c9eddba6513f: Status 404 returned error can't find the container with id ef9493c51ad465b720972aae2223b77bac43d786a99662db70b8c9eddba6513f
Oct 30 23:39:41.132550 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:41.132527    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:41.426004 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:41.425974    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"]
Oct 30 23:39:41.427465 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:41.427440    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb9dbde_ccfc_4d1b_852d_1e7203497c07.slice/crio-ceb602b36c763560da5280fb948f90d7dd6371c3421c879f0976e64ffbc493d8 WatchSource:0}: Error finding container ceb602b36c763560da5280fb948f90d7dd6371c3421c879f0976e64ffbc493d8: Status 404 returned error can't find the container with id ceb602b36c763560da5280fb948f90d7dd6371c3421c879f0976e64ffbc493d8
Oct 30 23:39:41.430033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:41.429998    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:39:41.430145 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:41.430080    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:39:41.430145 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:41.430127    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:39:42.034204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:42.034162    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n" event={"ID":"a4526b5d-ba8d-410e-8f1c-717ea858c74e","Type":"ContainerStarted","Data":"04c3c249dcda8d2948eb2573d6ae769309b34ed92b8a8ea7fbebf7df03c54577"}
Oct 30 23:39:42.034678 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:42.034257    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n" event={"ID":"a4526b5d-ba8d-410e-8f1c-717ea858c74e","Type":"ContainerStarted","Data":"ef9493c51ad465b720972aae2223b77bac43d786a99662db70b8c9eddba6513f"}
Oct 30 23:39:42.036146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:42.036116    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" event={"ID":"4bb9dbde-ccfc-4d1b-852d-1e7203497c07","Type":"ContainerStarted","Data":"5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b"}
Oct 30 23:39:42.036267 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:42.036169    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" event={"ID":"4bb9dbde-ccfc-4d1b-852d-1e7203497c07","Type":"ContainerStarted","Data":"ceb602b36c763560da5280fb948f90d7dd6371c3421c879f0976e64ffbc493d8"}
Oct 30 23:39:42.057908 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:42.057863    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n" podStartSLOduration=2.057849909 podStartE2EDuration="2.057849909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:42.055230803 +0000 UTC m=+1240.981868382" watchObservedRunningTime="2026-10-30 23:39:42.057849909 +0000 UTC m=+1240.984487489"
Oct 30 23:39:42.099397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:42.093323    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" podStartSLOduration=2.093299354 podStartE2EDuration="2.093299354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:42.081570674 +0000 UTC m=+1241.008208256" watchObservedRunningTime="2026-10-30 23:39:42.093299354 +0000 UTC m=+1241.019936937"
Oct 30 23:39:42.132788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:42.132757    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:42.134483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:42.134463    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:39:42.134546 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:42.134509    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 23:39:43.045723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:43.045666    4555 generic.go:356] "Generic (PLEG): container finished" podID="a4526b5d-ba8d-410e-8f1c-717ea858c74e" containerID="04c3c249dcda8d2948eb2573d6ae769309b34ed92b8a8ea7fbebf7df03c54577" exitCode=0
Oct 30 23:39:43.046409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:43.046293    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n" event={"ID":"a4526b5d-ba8d-410e-8f1c-717ea858c74e","Type":"ContainerDied","Data":"04c3c249dcda8d2948eb2573d6ae769309b34ed92b8a8ea7fbebf7df03c54577"}
Oct 30 23:39:43.133272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:43.133239    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:39:43.133451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:43.133297    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 23:39:44.003388 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.003323    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"]
Oct 30 23:39:44.007938 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.007922    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.025422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.024159    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"]
Oct 30 23:39:44.126129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.126097    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4570f25a-7dcwj\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.126560 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.126156    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l5wg\" (UniqueName: \"kubernetes.io/projected/5c267043-cb73-4967-ae79-d913a7d52942-kube-api-access-8l5wg\") pod \"maas-api-revoke-sub-4570f25a-7dcwj\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.126560 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.126273    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4570f25a-7dcwj\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.133560 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.133532    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:39:44.133719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.133580    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 23:39:44.227059 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.227009    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4570f25a-7dcwj\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.227233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.227105    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l5wg\" (UniqueName: \"kubernetes.io/projected/5c267043-cb73-4967-ae79-d913a7d52942-kube-api-access-8l5wg\") pod \"maas-api-revoke-sub-4570f25a-7dcwj\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.227233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.227184    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4570f25a-7dcwj\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.228070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.228045    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4570f25a-7dcwj\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.230306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.230283    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4570f25a-7dcwj\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.242685 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.242659    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l5wg\" (UniqueName: \"kubernetes.io/projected/5c267043-cb73-4967-ae79-d913a7d52942-kube-api-access-8l5wg\") pod \"maas-api-revoke-sub-4570f25a-7dcwj\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.299752 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.299726    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:44.322922 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.322877    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:44.429438 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.428757    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4526b5d-ba8d-410e-8f1c-717ea858c74e-kube-api-access-42gcl\" (UniqueName: \"kubernetes.io/projected/a4526b5d-ba8d-410e-8f1c-717ea858c74e-kube-api-access-42gcl\") pod \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") "
Oct 30 23:39:44.429438 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.428913    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-openshift-ca\") pod \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") "
Oct 30 23:39:44.429438 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.428955    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-xks-ca\") pod \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\" (UID: \"a4526b5d-ba8d-410e-8f1c-717ea858c74e\") "
Oct 30 23:39:44.429685 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.429548    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-openshift-ca" pod "a4526b5d-ba8d-410e-8f1c-717ea858c74e" (UID: "a4526b5d-ba8d-410e-8f1c-717ea858c74e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:39:44.432304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.432278    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-xks-ca" pod "a4526b5d-ba8d-410e-8f1c-717ea858c74e" (UID: "a4526b5d-ba8d-410e-8f1c-717ea858c74e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:39:44.432304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.432288    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4526b5d-ba8d-410e-8f1c-717ea858c74e-kube-api-access-42gcl" pod "a4526b5d-ba8d-410e-8f1c-717ea858c74e" (UID: "a4526b5d-ba8d-410e-8f1c-717ea858c74e"). InnerVolumeSpecName "kube-api-access-42gcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:39:44.530528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.530495    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:44.530528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.530530    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4526b5d-ba8d-410e-8f1c-717ea858c74e-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:44.530776 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.530540    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gcl\" (UniqueName: \"kubernetes.io/projected/a4526b5d-ba8d-410e-8f1c-717ea858c74e-kube-api-access-42gcl\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:44.640209 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:44.640178    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"]
Oct 30 23:39:44.641165 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:44.641132    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c267043_cb73_4967_ae79_d913a7d52942.slice/crio-bbab99fe55d41cbed4d1b3f16aed21fc5d0c65afa65ca27747bbd51c54cd61c1 WatchSource:0}: Error finding container bbab99fe55d41cbed4d1b3f16aed21fc5d0c65afa65ca27747bbd51c54cd61c1: Status 404 returned error can't find the container with id bbab99fe55d41cbed4d1b3f16aed21fc5d0c65afa65ca27747bbd51c54cd61c1
Oct 30 23:39:45.059763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:45.059729    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj" event={"ID":"5c267043-cb73-4967-ae79-d913a7d52942","Type":"ContainerStarted","Data":"bbab99fe55d41cbed4d1b3f16aed21fc5d0c65afa65ca27747bbd51c54cd61c1"}
Oct 30 23:39:45.061279 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:45.061255    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n" event={"ID":"a4526b5d-ba8d-410e-8f1c-717ea858c74e","Type":"ContainerDied","Data":"ef9493c51ad465b720972aae2223b77bac43d786a99662db70b8c9eddba6513f"}
Oct 30 23:39:45.061442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:45.061294    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef9493c51ad465b720972aae2223b77bac43d786a99662db70b8c9eddba6513f"
Oct 30 23:39:45.061442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:45.061295    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"
Oct 30 23:39:45.133520 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:45.133495    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:39:45.133858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:45.133543    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 23:39:46.055581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.055544    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"]
Oct 30 23:39:46.056081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.056066    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a4526b5d-ba8d-410e-8f1c-717ea858c74e" containerName="revoke-keys"
Oct 30 23:39:46.056121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.056083    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4526b5d-ba8d-410e-8f1c-717ea858c74e" containerName="revoke-keys"
Oct 30 23:39:46.056168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.056159    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4526b5d-ba8d-410e-8f1c-717ea858c74e" containerName="revoke-keys"
Oct 30 23:39:46.059483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.059464    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.083396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.081454    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"]
Oct 30 23:39:46.111689 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.111654    4555 generic.go:356] "Generic (PLEG): container finished" podID="5c267043-cb73-4967-ae79-d913a7d52942" containerID="04150b5bdb1e6d97a65df1526cc8080db147585cb838cfa4b4bc120d39acb59d" exitCode=0
Oct 30 23:39:46.111857 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.111704    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj" event={"ID":"5c267043-cb73-4967-ae79-d913a7d52942","Type":"ContainerDied","Data":"04150b5bdb1e6d97a65df1526cc8080db147585cb838cfa4b4bc120d39acb59d"}
Oct 30 23:39:46.141326 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.141294    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:39:46.141889 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.141850    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 23:39:46.147671 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.147646    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6124650-76hzr\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.147821 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.147702    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zq2n\" (UniqueName: \"kubernetes.io/projected/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-kube-api-access-2zq2n\") pod \"maas-api-revoke-sub-c6124650-76hzr\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.147821 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.147809    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6124650-76hzr\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.255789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.249964    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6124650-76hzr\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.255789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.250049    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zq2n\" (UniqueName: \"kubernetes.io/projected/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-kube-api-access-2zq2n\") pod \"maas-api-revoke-sub-c6124650-76hzr\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.255789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.250145    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6124650-76hzr\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.255789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.250983    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6124650-76hzr\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.280396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.277978    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zq2n\" (UniqueName: \"kubernetes.io/projected/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-kube-api-access-2zq2n\") pod \"maas-api-revoke-sub-c6124650-76hzr\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.280571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.280454    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6124650-76hzr\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.413438 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.412823    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:46.759757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:46.759731    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"]
Oct 30 23:39:46.761821 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:46.761790    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2005ffc_37a6_4430_b9ef_632d1dbb28d2.slice/crio-885cce16835cf6a5bbb2740e4004ec5a45bcba9efc3212c0b726a07a075cbff8 WatchSource:0}: Error finding container 885cce16835cf6a5bbb2740e4004ec5a45bcba9efc3212c0b726a07a075cbff8: Status 404 returned error can't find the container with id 885cce16835cf6a5bbb2740e4004ec5a45bcba9efc3212c0b726a07a075cbff8
Oct 30 23:39:47.120033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.119997    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr" event={"ID":"f2005ffc-37a6-4430-b9ef-632d1dbb28d2","Type":"ContainerStarted","Data":"885cce16835cf6a5bbb2740e4004ec5a45bcba9efc3212c0b726a07a075cbff8"}
Oct 30 23:39:47.133713 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.133684    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:39:47.133855 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.133741    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 23:39:47.353777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.353748    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:47.505601 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.505506    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-openshift-ca\") pod \"5c267043-cb73-4967-ae79-d913a7d52942\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") "
Oct 30 23:39:47.505792 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.505633    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-xks-ca\") pod \"5c267043-cb73-4967-ae79-d913a7d52942\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") "
Oct 30 23:39:47.505792 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.505760    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c267043-cb73-4967-ae79-d913a7d52942-kube-api-access-8l5wg\" (UniqueName: \"kubernetes.io/projected/5c267043-cb73-4967-ae79-d913a7d52942-kube-api-access-8l5wg\") pod \"5c267043-cb73-4967-ae79-d913a7d52942\" (UID: \"5c267043-cb73-4967-ae79-d913a7d52942\") "
Oct 30 23:39:47.506413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.506346    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-openshift-ca" pod "5c267043-cb73-4967-ae79-d913a7d52942" (UID: "5c267043-cb73-4967-ae79-d913a7d52942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:39:47.508265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.508238    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-xks-ca" pod "5c267043-cb73-4967-ae79-d913a7d52942" (UID: "5c267043-cb73-4967-ae79-d913a7d52942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:39:47.508855 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.508831    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c267043-cb73-4967-ae79-d913a7d52942-kube-api-access-8l5wg" pod "5c267043-cb73-4967-ae79-d913a7d52942" (UID: "5c267043-cb73-4967-ae79-d913a7d52942"). InnerVolumeSpecName "kube-api-access-8l5wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:39:47.607494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.607460    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:47.607494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.607494    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c267043-cb73-4967-ae79-d913a7d52942-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:47.607723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:47.607507    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l5wg\" (UniqueName: \"kubernetes.io/projected/5c267043-cb73-4967-ae79-d913a7d52942-kube-api-access-8l5wg\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:48.129407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:48.129296    4555 generic.go:356] "Generic (PLEG): container finished" podID="f2005ffc-37a6-4430-b9ef-632d1dbb28d2" containerID="8d4e78ed36f7d3107bee6bcb0f4d42486735f4c67f408ab0b55206bee913339e" exitCode=0
Oct 30 23:39:48.129580 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:48.129409    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr" event={"ID":"f2005ffc-37a6-4430-b9ef-632d1dbb28d2","Type":"ContainerDied","Data":"8d4e78ed36f7d3107bee6bcb0f4d42486735f4c67f408ab0b55206bee913339e"}
Oct 30 23:39:48.130945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:48.130922    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"
Oct 30 23:39:48.130945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:48.130931    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj" event={"ID":"5c267043-cb73-4967-ae79-d913a7d52942","Type":"ContainerDied","Data":"bbab99fe55d41cbed4d1b3f16aed21fc5d0c65afa65ca27747bbd51c54cd61c1"}
Oct 30 23:39:48.131154 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:48.130972    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbab99fe55d41cbed4d1b3f16aed21fc5d0c65afa65ca27747bbd51c54cd61c1"
Oct 30 23:39:49.109636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.109064    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:49.110117 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.110084    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:49.111979 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.111952    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:39:49.301569 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.301546    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:49.429477 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.429357    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-xks-ca\") pod \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") "
Oct 30 23:39:49.429660 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.429492    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-openshift-ca\") pod \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") "
Oct 30 23:39:49.429660 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.429517    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-kube-api-access-2zq2n\" (UniqueName: \"kubernetes.io/projected/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-kube-api-access-2zq2n\") pod \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\" (UID: \"f2005ffc-37a6-4430-b9ef-632d1dbb28d2\") "
Oct 30 23:39:49.429881 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.429853    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-openshift-ca" pod "f2005ffc-37a6-4430-b9ef-632d1dbb28d2" (UID: "f2005ffc-37a6-4430-b9ef-632d1dbb28d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:39:49.429980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.429950    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:49.431808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.431783    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-kube-api-access-2zq2n" pod "f2005ffc-37a6-4430-b9ef-632d1dbb28d2" (UID: "f2005ffc-37a6-4430-b9ef-632d1dbb28d2"). InnerVolumeSpecName "kube-api-access-2zq2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:39:49.431950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.431934    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-xks-ca" pod "f2005ffc-37a6-4430-b9ef-632d1dbb28d2" (UID: "f2005ffc-37a6-4430-b9ef-632d1dbb28d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:39:49.531398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.531342    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zq2n\" (UniqueName: \"kubernetes.io/projected/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-kube-api-access-2zq2n\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:49.531398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:49.531390    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2005ffc-37a6-4430-b9ef-632d1dbb28d2-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:50.144025 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:50.143989    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"
Oct 30 23:39:50.144025 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:50.144013    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr" event={"ID":"f2005ffc-37a6-4430-b9ef-632d1dbb28d2","Type":"ContainerDied","Data":"885cce16835cf6a5bbb2740e4004ec5a45bcba9efc3212c0b726a07a075cbff8"}
Oct 30 23:39:50.144582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:50.144066    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="885cce16835cf6a5bbb2740e4004ec5a45bcba9efc3212c0b726a07a075cbff8"
Oct 30 23:39:55.916288 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:55.916252    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"]
Oct 30 23:39:55.916728 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:55.916713    4555 cpu_manager.go:414] "containerMap: removing container" podUID="5c267043-cb73-4967-ae79-d913a7d52942" containerName="revoke-keys"
Oct 30 23:39:55.916771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:55.916729    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c267043-cb73-4967-ae79-d913a7d52942" containerName="revoke-keys"
Oct 30 23:39:55.916771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:55.916757    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f2005ffc-37a6-4430-b9ef-632d1dbb28d2" containerName="revoke-keys"
Oct 30 23:39:55.916771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:55.916762    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2005ffc-37a6-4430-b9ef-632d1dbb28d2" containerName="revoke-keys"
Oct 30 23:39:55.916878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:55.916831    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2005ffc-37a6-4430-b9ef-632d1dbb28d2" containerName="revoke-keys"
Oct 30 23:39:55.916878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:55.916841    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c267043-cb73-4967-ae79-d913a7d52942" containerName="revoke-keys"
Oct 30 23:39:55.920236 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:55.920216    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:55.928634 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:55.928610    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e15304-openshift-default-dockercfg-qbhbd\""
Oct 30 23:39:55.936921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:55.936894    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"]
Oct 30 23:39:56.022307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.022269    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdqc\" (UniqueName: \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-kube-api-access-nhdqc\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.022499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.022319    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8aee7e5-9755-4dee-8000-31115262a4c3-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.022499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.022412    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.022617 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.022501    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.022617 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.022549    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.022744 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.022635    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.022744 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.022681    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.022823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.022740    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.022859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.022821    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.123553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.123515    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8aee7e5-9755-4dee-8000-31115262a4c3-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.123553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.123560    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.123763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.123590    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.123763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.123622    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.123763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.123650    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.123763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.123686    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.123763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.123742    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.123966 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.123816    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.123966 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.123851    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdqc\" (UniqueName: \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-kube-api-access-nhdqc\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.123966 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.123936    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.124291 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.124272    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.124359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.124319    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.124443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.124422    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.124620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.124602    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8aee7e5-9755-4dee-8000-31115262a4c3-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.126506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.126487    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.126566 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.126485    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.135883 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.135860    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.136094 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.136070    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdqc\" (UniqueName: \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-kube-api-access-nhdqc\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.235160 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.235080    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:56.375089 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.375054    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"]
Oct 30 23:39:56.379614 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.379590    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.388729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.388693    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:39:56.390799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.390767    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"]
Oct 30 23:39:56.391690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.391669    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:39:56.427632 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.427607    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrrjx\" (UniqueName: \"kubernetes.io/projected/9837086d-b44c-4638-9a48-57d10b590c32-kube-api-access-jrrjx\") pod \"maas-api-revoke-sub-5ef13daf-c6bgf\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.427791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.427681    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef13daf-c6bgf\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.427791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.427748    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef13daf-c6bgf\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.523734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.523708    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"]
Oct 30 23:39:56.526083 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:56.526051    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8aee7e5_9755_4dee_8000_31115262a4c3.slice/crio-72734eb11b4df5019c4e3de0c133b310d6e6831628fba6bef7228420bf78e4fb WatchSource:0}: Error finding container 72734eb11b4df5019c4e3de0c133b310d6e6831628fba6bef7228420bf78e4fb: Status 404 returned error can't find the container with id 72734eb11b4df5019c4e3de0c133b310d6e6831628fba6bef7228420bf78e4fb
Oct 30 23:39:56.528510 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.528483    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrrjx\" (UniqueName: \"kubernetes.io/projected/9837086d-b44c-4638-9a48-57d10b590c32-kube-api-access-jrrjx\") pod \"maas-api-revoke-sub-5ef13daf-c6bgf\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.528607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.528592    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef13daf-c6bgf\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.528650 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.528621    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef13daf-c6bgf\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.528650 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.528613    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:39:56.528731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.528686    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:39:56.528764 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.528730    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:39:56.529733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.529399    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef13daf-c6bgf\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.531541 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.531523    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef13daf-c6bgf\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.541086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.541061    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrrjx\" (UniqueName: \"kubernetes.io/projected/9837086d-b44c-4638-9a48-57d10b590c32-kube-api-access-jrrjx\") pod \"maas-api-revoke-sub-5ef13daf-c6bgf\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.715448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.715416    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:56.918527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.918486    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"]
Oct 30 23:39:56.924179 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.924151    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:56.929122 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:56.929091    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"]
Oct 30 23:39:57.010509 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.010483    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"]
Oct 30 23:39:57.012258 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:57.012234    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9837086d_b44c_4638_9a48_57d10b590c32.slice/crio-cb54179b1b55332df724c855bbf88ba3f1cbb1088abfac72936cf8e146208d05 WatchSource:0}: Error finding container cb54179b1b55332df724c855bbf88ba3f1cbb1088abfac72936cf8e146208d05: Status 404 returned error can't find the container with id cb54179b1b55332df724c855bbf88ba3f1cbb1088abfac72936cf8e146208d05
Oct 30 23:39:57.033426 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.033403    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732b2e61-7dx2d\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:57.033547 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.033529    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5jt\" (UniqueName: \"kubernetes.io/projected/740f8605-1c38-4d02-9ebd-1c4101193dbc-kube-api-access-xk5jt\") pod \"maas-api-revoke-sub-732b2e61-7dx2d\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:57.033590 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.033560    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732b2e61-7dx2d\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:57.134610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.134579    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732b2e61-7dx2d\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:57.134783 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.134673    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5jt\" (UniqueName: \"kubernetes.io/projected/740f8605-1c38-4d02-9ebd-1c4101193dbc-kube-api-access-xk5jt\") pod \"maas-api-revoke-sub-732b2e61-7dx2d\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:57.134783 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.134694    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732b2e61-7dx2d\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:57.135332 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.135306    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732b2e61-7dx2d\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:57.137248 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.137221    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732b2e61-7dx2d\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:57.147589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.147561    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5jt\" (UniqueName: \"kubernetes.io/projected/740f8605-1c38-4d02-9ebd-1c4101193dbc-kube-api-access-xk5jt\") pod \"maas-api-revoke-sub-732b2e61-7dx2d\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:57.208010 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.207969    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" event={"ID":"b8aee7e5-9755-4dee-8000-31115262a4c3","Type":"ContainerStarted","Data":"623aac7033a22e651ef10b7e7d067c2f121f04f60813a5ee1dcd35b87ad2447b"}
Oct 30 23:39:57.208203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.208020    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" event={"ID":"b8aee7e5-9755-4dee-8000-31115262a4c3","Type":"ContainerStarted","Data":"72734eb11b4df5019c4e3de0c133b310d6e6831628fba6bef7228420bf78e4fb"}
Oct 30 23:39:57.208203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.208037    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf" event={"ID":"9837086d-b44c-4638-9a48-57d10b590c32","Type":"ContainerStarted","Data":"cb54179b1b55332df724c855bbf88ba3f1cbb1088abfac72936cf8e146208d05"}
Oct 30 23:39:57.227012 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.226960    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podStartSLOduration=2.226946757 podStartE2EDuration="2.226946757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:57.223907399 +0000 UTC m=+1256.150544973" watchObservedRunningTime="2026-10-30 23:39:57.226946757 +0000 UTC m=+1256.153584337"
Oct 30 23:39:57.235670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.235579    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:39:57.237068 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.237044    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:39:57.237215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.237096    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:39:57.243790 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.243770    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:39:57.460231 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:39:57.460196    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:39:57.532404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:57.532358    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"]
Oct 30 23:39:57.533707 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:39:57.533682    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice/crio-5a3eb277b0a4457447076a989ff75ca40a97d3fe8c290643a8d0e2c0a389eb11 WatchSource:0}: Error finding container 5a3eb277b0a4457447076a989ff75ca40a97d3fe8c290643a8d0e2c0a389eb11: Status 404 returned error can't find the container with id 5a3eb277b0a4457447076a989ff75ca40a97d3fe8c290643a8d0e2c0a389eb11
Oct 30 23:39:58.213952 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:58.213914    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d" event={"ID":"740f8605-1c38-4d02-9ebd-1c4101193dbc","Type":"ContainerStarted","Data":"20b7b4f9f260671f846b3d8e13608bd6ea4ac6649d8833569e73ef05a092dc91"}
Oct 30 23:39:58.214477 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:58.213982    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d" event={"ID":"740f8605-1c38-4d02-9ebd-1c4101193dbc","Type":"ContainerStarted","Data":"5a3eb277b0a4457447076a989ff75ca40a97d3fe8c290643a8d0e2c0a389eb11"}
Oct 30 23:39:58.215893 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:58.215864    4555 generic.go:356] "Generic (PLEG): container finished" podID="9837086d-b44c-4638-9a48-57d10b590c32" containerID="23b90a949d4d4917dff9a66db01353c1a9eb314db66ac524e7d3a05a8068cb12" exitCode=0
Oct 30 23:39:58.216051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:58.215943    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf" event={"ID":"9837086d-b44c-4638-9a48-57d10b590c32","Type":"ContainerDied","Data":"23b90a949d4d4917dff9a66db01353c1a9eb314db66ac524e7d3a05a8068cb12"}
Oct 30 23:39:58.234416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:58.234324    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d" podStartSLOduration=2.234307338 podStartE2EDuration="2.234307338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:58.231791407 +0000 UTC m=+1257.158428989" watchObservedRunningTime="2026-10-30 23:39:58.234307338 +0000 UTC m=+1257.160944921"
Oct 30 23:39:58.235544 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:58.235509    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:39:58.235662 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:58.235554    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:39:59.148521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.148479    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:39:59.225172 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.225122    4555 generic.go:356] "Generic (PLEG): container finished" podID="740f8605-1c38-4d02-9ebd-1c4101193dbc" containerID="20b7b4f9f260671f846b3d8e13608bd6ea4ac6649d8833569e73ef05a092dc91" exitCode=0
Oct 30 23:39:59.225690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.225459    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d" event={"ID":"740f8605-1c38-4d02-9ebd-1c4101193dbc","Type":"ContainerDied","Data":"20b7b4f9f260671f846b3d8e13608bd6ea4ac6649d8833569e73ef05a092dc91"}
Oct 30 23:39:59.235478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.235449    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:39:59.235622 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.235502    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:39:59.495459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.475821    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:39:59.587397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.586720    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-xks-ca\") pod \"9837086d-b44c-4638-9a48-57d10b590c32\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") "
Oct 30 23:39:59.587397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.586781    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-openshift-ca\") pod \"9837086d-b44c-4638-9a48-57d10b590c32\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") "
Oct 30 23:39:59.587397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.586858    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9837086d-b44c-4638-9a48-57d10b590c32-kube-api-access-jrrjx\" (UniqueName: \"kubernetes.io/projected/9837086d-b44c-4638-9a48-57d10b590c32-kube-api-access-jrrjx\") pod \"9837086d-b44c-4638-9a48-57d10b590c32\" (UID: \"9837086d-b44c-4638-9a48-57d10b590c32\") "
Oct 30 23:39:59.592397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.588714    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-openshift-ca" pod "9837086d-b44c-4638-9a48-57d10b590c32" (UID: "9837086d-b44c-4638-9a48-57d10b590c32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:39:59.594967 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.594898    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-xks-ca" pod "9837086d-b44c-4638-9a48-57d10b590c32" (UID: "9837086d-b44c-4638-9a48-57d10b590c32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:39:59.594967 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.594895    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9837086d-b44c-4638-9a48-57d10b590c32-kube-api-access-jrrjx" pod "9837086d-b44c-4638-9a48-57d10b590c32" (UID: "9837086d-b44c-4638-9a48-57d10b590c32"). InnerVolumeSpecName "kube-api-access-jrrjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:39:59.688453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.688127    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:59.688453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.688178    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrrjx\" (UniqueName: \"kubernetes.io/projected/9837086d-b44c-4638-9a48-57d10b590c32-kube-api-access-jrrjx\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:59.688453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:39:59.688195    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9837086d-b44c-4638-9a48-57d10b590c32-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:40:00.269272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:00.265209    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:40:00.269272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:00.265267    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:40:00.273050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:00.273021    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"
Oct 30 23:40:00.276598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:00.276573    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf" event={"ID":"9837086d-b44c-4638-9a48-57d10b590c32","Type":"ContainerDied","Data":"cb54179b1b55332df724c855bbf88ba3f1cbb1088abfac72936cf8e146208d05"}
Oct 30 23:40:00.276815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:00.276798    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb54179b1b55332df724c855bbf88ba3f1cbb1088abfac72936cf8e146208d05"
Oct 30 23:40:00.936695 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:00.936222    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:40:01.131003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.130175    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-xks-ca\") pod \"740f8605-1c38-4d02-9ebd-1c4101193dbc\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") "
Oct 30 23:40:01.131003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.130245    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-openshift-ca\") pod \"740f8605-1c38-4d02-9ebd-1c4101193dbc\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") "
Oct 30 23:40:01.131003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.130288    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740f8605-1c38-4d02-9ebd-1c4101193dbc-kube-api-access-xk5jt\" (UniqueName: \"kubernetes.io/projected/740f8605-1c38-4d02-9ebd-1c4101193dbc-kube-api-access-xk5jt\") pod \"740f8605-1c38-4d02-9ebd-1c4101193dbc\" (UID: \"740f8605-1c38-4d02-9ebd-1c4101193dbc\") "
Oct 30 23:40:01.171854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.171817    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-openshift-ca" pod "740f8605-1c38-4d02-9ebd-1c4101193dbc" (UID: "740f8605-1c38-4d02-9ebd-1c4101193dbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:40:01.194549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.194267    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-xks-ca" pod "740f8605-1c38-4d02-9ebd-1c4101193dbc" (UID: "740f8605-1c38-4d02-9ebd-1c4101193dbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:40:01.212396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.198749    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740f8605-1c38-4d02-9ebd-1c4101193dbc-kube-api-access-xk5jt" pod "740f8605-1c38-4d02-9ebd-1c4101193dbc" (UID: "740f8605-1c38-4d02-9ebd-1c4101193dbc"). InnerVolumeSpecName "kube-api-access-xk5jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:01.243586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.234280    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:40:01.243586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.234310    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740f8605-1c38-4d02-9ebd-1c4101193dbc-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:40:01.243586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.234327    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk5jt\" (UniqueName: \"kubernetes.io/projected/740f8605-1c38-4d02-9ebd-1c4101193dbc-kube-api-access-xk5jt\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:40:01.257394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.244236    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:40:01.257394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.244291    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:40:01.401437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.389477    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d" event={"ID":"740f8605-1c38-4d02-9ebd-1c4101193dbc","Type":"ContainerDied","Data":"5a3eb277b0a4457447076a989ff75ca40a97d3fe8c290643a8d0e2c0a389eb11"}
Oct 30 23:40:01.401437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.389541    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a3eb277b0a4457447076a989ff75ca40a97d3fe8c290643a8d0e2c0a389eb11"
Oct 30 23:40:01.401437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:01.389614    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"
Oct 30 23:40:02.236308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:02.236245    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:40:02.236675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:02.236324    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:40:03.235965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:03.235936    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:40:03.236343 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:03.235984    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:40:04.678784 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:04.678742    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:40:04.679950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:04.679927    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:40:04.680728 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:04.680699    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:40:05.917642 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:05.917603    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:06.211359 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:06.211277    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:10.594133 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.594097    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"]
Oct 30 23:40:10.596615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.594878    4555 cpu_manager.go:414] "containerMap: removing container" podUID="740f8605-1c38-4d02-9ebd-1c4101193dbc" containerName="revoke-keys"
Oct 30 23:40:10.596615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.594899    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740f8605-1c38-4d02-9ebd-1c4101193dbc" containerName="revoke-keys"
Oct 30 23:40:10.596615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.594929    4555 cpu_manager.go:414] "containerMap: removing container" podUID="9837086d-b44c-4638-9a48-57d10b590c32" containerName="revoke-keys"
Oct 30 23:40:10.596615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.594938    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9837086d-b44c-4638-9a48-57d10b590c32" containerName="revoke-keys"
Oct 30 23:40:10.596615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.595072    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="740f8605-1c38-4d02-9ebd-1c4101193dbc" containerName="revoke-keys"
Oct 30 23:40:10.596615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.595087    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="9837086d-b44c-4638-9a48-57d10b590c32" containerName="revoke-keys"
Oct 30 23:40:10.598767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.598741    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.615219 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.615191    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"]
Oct 30 23:40:10.622007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.621979    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-611ef3\"/\"distinct-w1-611ef3-kserve-self-signed-certs\""
Oct 30 23:40:10.651810 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.651777    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-dshm\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.651949 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.651882    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-tmp-dir\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.652030 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.651959    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-model-cache\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.652126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.652099    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-home\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.652267 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.652243    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslnd\" (UniqueName: \"kubernetes.io/projected/3b0a5c64-d800-4ce5-bfdb-743907777d34-kube-api-access-vslnd\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.652337 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.652311    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0a5c64-d800-4ce5-bfdb-743907777d34-tls-certs\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.753778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.753725    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-model-cache\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.754103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.754084    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-home\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.754329 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.754312    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslnd\" (UniqueName: \"kubernetes.io/projected/3b0a5c64-d800-4ce5-bfdb-743907777d34-kube-api-access-vslnd\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.754516 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.754492    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0a5c64-d800-4ce5-bfdb-743907777d34-tls-certs\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.754695 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.754663    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-dshm\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.754841 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.754825    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-tmp-dir\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.755542 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.755512    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-tmp-dir\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.758026 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.758006    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-model-cache\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.759042 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.758892    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-home\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.770998 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.768676    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-dshm\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.770998 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.770536    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0a5c64-d800-4ce5-bfdb-743907777d34-tls-certs\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.810159 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.810121    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslnd\" (UniqueName: \"kubernetes.io/projected/3b0a5c64-d800-4ce5-bfdb-743907777d34-kube-api-access-vslnd\") pod \"distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") " pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:10.926664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:10.926570    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:11.236805 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:11.236780    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"]
Oct 30 23:40:11.238934 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:40:11.238891    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b0a5c64_d800_4ce5_bfdb_743907777d34.slice/crio-d1c7b2212898cebbf9faf93b24d9cc06dd87b451dd14ecdbc74e766f16d9b89c WatchSource:0}: Error finding container d1c7b2212898cebbf9faf93b24d9cc06dd87b451dd14ecdbc74e766f16d9b89c: Status 404 returned error can't find the container with id d1c7b2212898cebbf9faf93b24d9cc06dd87b451dd14ecdbc74e766f16d9b89c
Oct 30 23:40:11.478564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:11.478524    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" event={"ID":"3b0a5c64-d800-4ce5-bfdb-743907777d34","Type":"ContainerStarted","Data":"c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52"}
Oct 30 23:40:11.478564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:11.478564    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" event={"ID":"3b0a5c64-d800-4ce5-bfdb-743907777d34","Type":"ContainerStarted","Data":"d1c7b2212898cebbf9faf93b24d9cc06dd87b451dd14ecdbc74e766f16d9b89c"}
Oct 30 23:40:11.502340 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:11.502245    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podStartSLOduration=1.502231557 podStartE2EDuration="1.502231557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:11.50121704 +0000 UTC m=+1270.427854621" watchObservedRunningTime="2026-10-30 23:40:11.502231557 +0000 UTC m=+1270.428869163"
Oct 30 23:40:11.926807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:11.926777    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:15.976765 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:15.976730    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:18.747811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.747772    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"]
Oct 30 23:40:18.753520 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.753493    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.760662 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.760638    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"openshift-service-ca.crt\""
Oct 30 23:40:18.760957 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.760940    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"kube-root-ca.crt\""
Oct 30 23:40:18.765392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.761442    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"distinct-w3-b567bb-kserve-self-signed-certs\""
Oct 30 23:40:18.807445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.761468    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b567bb\"/\"default-dockercfg-jtbmr\""
Oct 30 23:40:18.807445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.764527    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"]
Oct 30 23:40:18.866407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.866340    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-home\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.866595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.866414    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-dshm\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.866595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.866448    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684d4151-975d-49f1-90d5-84a476599936-tls-certs\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.866595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.866524    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-model-cache\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.866595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.866557    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-tmp-dir\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.866803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.866699    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2k4m\" (UniqueName: \"kubernetes.io/projected/684d4151-975d-49f1-90d5-84a476599936-kube-api-access-d2k4m\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.970407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.968101    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-home\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.970407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.968161    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-dshm\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.970407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.968190    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684d4151-975d-49f1-90d5-84a476599936-tls-certs\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.970407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.968262    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-model-cache\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.970407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.968295    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-tmp-dir\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.970407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.968481    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2k4m\" (UniqueName: \"kubernetes.io/projected/684d4151-975d-49f1-90d5-84a476599936-kube-api-access-d2k4m\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.970407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.969360    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-model-cache\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.970407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.969622    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-home\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.970407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.970218    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-tmp-dir\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.987396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.979017    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-dshm\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.991397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.988947    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684d4151-975d-49f1-90d5-84a476599936-tls-certs\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:18.996001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:18.994108    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2k4m\" (UniqueName: \"kubernetes.io/projected/684d4151-975d-49f1-90d5-84a476599936-kube-api-access-d2k4m\") pod \"distinct-w3-b567bb-kserve-656b4f7bf6-lxskl\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") " pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:19.082399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:19.080553    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:19.386529 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:19.386498    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"]
Oct 30 23:40:19.388194 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:40:19.388161    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod684d4151_975d_49f1_90d5_84a476599936.slice/crio-82ce2327838ba289cb82fe0e8b7d0eda171b6e56772b624d73eb40c2d0104087 WatchSource:0}: Error finding container 82ce2327838ba289cb82fe0e8b7d0eda171b6e56772b624d73eb40c2d0104087: Status 404 returned error can't find the container with id 82ce2327838ba289cb82fe0e8b7d0eda171b6e56772b624d73eb40c2d0104087
Oct 30 23:40:19.561033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:19.560435    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" event={"ID":"684d4151-975d-49f1-90d5-84a476599936","Type":"ContainerStarted","Data":"04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8"}
Oct 30 23:40:19.561033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:19.560477    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" event={"ID":"684d4151-975d-49f1-90d5-84a476599936","Type":"ContainerStarted","Data":"82ce2327838ba289cb82fe0e8b7d0eda171b6e56772b624d73eb40c2d0104087"}
Oct 30 23:40:19.591840 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:19.591770    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podStartSLOduration=1.5917492389999999 podStartE2EDuration="1.591749239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:19.58767589 +0000 UTC m=+1278.514313472" watchObservedRunningTime="2026-10-30 23:40:19.591749239 +0000 UTC m=+1278.518386821"
Oct 30 23:40:20.081869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:20.081824    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:20.926811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:20.926774    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:20.944855 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:20.944827    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:21.255465 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:21.255392    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:21.602494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:21.602452    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:40:25.574133 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.574094    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"]
Oct 30 23:40:25.579838 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.579808    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.587387 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.587330    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9ef03a-openshift-default-dockercfg-mf4wj\""
Oct 30 23:40:25.596309 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.596278    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"]
Oct 30 23:40:25.625195 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.625161    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"]
Oct 30 23:40:25.629320 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.629295    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.643663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.643030    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"]
Oct 30 23:40:25.659446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.659412    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.659607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.659458    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.659607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.659570    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.659607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.659600    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzfl\" (UniqueName: \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-kube-api-access-8wzfl\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.659762 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.659623    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.659762 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.659752    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.659851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.659814    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.659904 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.659870    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.660002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.659984    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88785011-298c-4593-9d9e-cc6b0c8f5297-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.687712 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.687680    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-843d8c-openshift-default-dockercfg-l4st5\""
Oct 30 23:40:25.761684 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.761649    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.761684 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.761686    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.761941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.761711    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.761941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.761760    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.761941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.761793    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.761941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.761828    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.761941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.761860    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn4kh\" (UniqueName: \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-kube-api-access-pn4kh\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.761941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.761900    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.761941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.761925    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.762334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.761958    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a692a4-2c57-422b-8f8e-8e8b206a0232-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.762334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762000    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.762334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762067    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.762334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762158    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.762334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762183    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.762334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762266    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88785011-298c-4593-9d9e-cc6b0c8f5297-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.762334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762312    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.762705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762547    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.762705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762573    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.762705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762599    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.762705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762630    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.762705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762663    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzfl\" (UniqueName: \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-kube-api-access-8wzfl\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.762974 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.762940    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88785011-298c-4593-9d9e-cc6b0c8f5297-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.763152 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.763131    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.765967 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.765941    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.767607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.767580    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.773876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.773852    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.775417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.775395    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzfl\" (UniqueName: \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-kube-api-access-8wzfl\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.860966 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.860870    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"]
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.864078    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn4kh\" (UniqueName: \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-kube-api-access-pn4kh\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.864141    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.864175    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a692a4-2c57-422b-8f8e-8e8b206a0232-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.864218    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.864263    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.864420    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.864482    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.864560    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.864601    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.865067    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.866663    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a692a4-2c57-422b-8f8e-8e8b206a0232-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.866925    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.867779 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.867177    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.869311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.868964    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.869311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.869243    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.870143 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.870116    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.870875 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.870856    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:25.890856 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.890814    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"]
Oct 30 23:40:25.895859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.895809    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn4kh\" (UniqueName: \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-kube-api-access-pn4kh\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.897088 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.896095    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.901241 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.900925    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:25.950552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.950518    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:25.966160 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.966131    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snjx2\" (UniqueName: \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-kube-api-access-snjx2\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:25.966282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.966210    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:25.966282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.966251    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70538c08-5ce2-41fa-ac84-d6427beb113b-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:25.966414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.966326    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:25.966414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.966350    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:25.966522 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.966468    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:25.966690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.966521    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:25.966690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.966539    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:25.966690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:25.966554    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:25.991879 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:25.990656    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:26.068149 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068104    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.068759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068195    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.068759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068222    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.068759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068245    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.068759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068326    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snjx2\" (UniqueName: \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-kube-api-access-snjx2\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.068759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068431    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.068759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068454    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70538c08-5ce2-41fa-ac84-d6427beb113b-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.068759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068495    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.068759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068527    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.068759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068647    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.069342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.068862    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.069342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.069115    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.069604 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.069583    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.070049 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.070029    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70538c08-5ce2-41fa-ac84-d6427beb113b-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.073191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.073162    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.073573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.073551    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.082205 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.082151    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snjx2\" (UniqueName: \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-kube-api-access-snjx2\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.083283 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.083236    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.213762 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:40:26.213726    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88785011_298c_4593_9d9e_cc6b0c8f5297.slice/crio-0dddf4b881e14d66d0a7d85f11c22a3092d379f8f13302278b64d9e7fdf56a50 WatchSource:0}: Error finding container 0dddf4b881e14d66d0a7d85f11c22a3092d379f8f13302278b64d9e7fdf56a50: Status 404 returned error can't find the container with id 0dddf4b881e14d66d0a7d85f11c22a3092d379f8f13302278b64d9e7fdf56a50
Oct 30 23:40:26.214651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.214557    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"]
Oct 30 23:40:26.216933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.216890    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:26.217070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.216990    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:26.217070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.217036    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:26.232683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.232657    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:26.291667 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:40:26.287591    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a692a4_2c57_422b_8f8e_8e8b206a0232.slice/crio-a5b39f5b9f268bdd2ac0392d308a9e24d7b1e744823a2970caf7cfb310ec74a0 WatchSource:0}: Error finding container a5b39f5b9f268bdd2ac0392d308a9e24d7b1e744823a2970caf7cfb310ec74a0: Status 404 returned error can't find the container with id a5b39f5b9f268bdd2ac0392d308a9e24d7b1e744823a2970caf7cfb310ec74a0
Oct 30 23:40:26.291667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.291124    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"]
Oct 30 23:40:26.295465 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.295427    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:26.295579 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.295509    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:26.295579 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.295548    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:26.606512 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.606348    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"]
Oct 30 23:40:26.615736 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:40:26.615687    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70538c08_5ce2_41fa_ac84_d6427beb113b.slice/crio-3d0dbb8499d9b0818d4aaab2ad174e3e11f121a37c6cc2747eec7780051058dc WatchSource:0}: Error finding container 3d0dbb8499d9b0818d4aaab2ad174e3e11f121a37c6cc2747eec7780051058dc: Status 404 returned error can't find the container with id 3d0dbb8499d9b0818d4aaab2ad174e3e11f121a37c6cc2747eec7780051058dc
Oct 30 23:40:26.631203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.630172    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" event={"ID":"99a692a4-2c57-422b-8f8e-8e8b206a0232","Type":"ContainerStarted","Data":"8a046115cddb657a92d11791a8f8317c10853353454dd4e752a1512088ccbf94"}
Oct 30 23:40:26.631203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.630243    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" event={"ID":"99a692a4-2c57-422b-8f8e-8e8b206a0232","Type":"ContainerStarted","Data":"a5b39f5b9f268bdd2ac0392d308a9e24d7b1e744823a2970caf7cfb310ec74a0"}
Oct 30 23:40:26.633929 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.633873    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" event={"ID":"88785011-298c-4593-9d9e-cc6b0c8f5297","Type":"ContainerStarted","Data":"7de9401a7a9db0f0a196272a9934fbcfff026a9634e8efe1fb7da4ebc282651a"}
Oct 30 23:40:26.634108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.634091    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" event={"ID":"88785011-298c-4593-9d9e-cc6b0c8f5297","Type":"ContainerStarted","Data":"0dddf4b881e14d66d0a7d85f11c22a3092d379f8f13302278b64d9e7fdf56a50"}
Oct 30 23:40:26.638778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.638740    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:26.638882 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.638823    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:26.638882 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.638868    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:26.691429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.688929    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" podStartSLOduration=1.6889107920000002 podStartE2EDuration="1.688910792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:26.688335328 +0000 UTC m=+1285.614972915" watchObservedRunningTime="2026-10-30 23:40:26.688910792 +0000 UTC m=+1285.615548407"
Oct 30 23:40:26.693482 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.693433    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" podStartSLOduration=1.693414936 podStartE2EDuration="1.693414936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:26.663259958 +0000 UTC m=+1285.589897540" watchObservedRunningTime="2026-10-30 23:40:26.693414936 +0000 UTC m=+1285.620052517"
Oct 30 23:40:26.902397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.901398    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:26.950976 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:26.950889    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:27.555429 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:27.554250    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:40:27.657856 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:27.657825    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" event={"ID":"70538c08-5ce2-41fa-ac84-d6427beb113b","Type":"ContainerStarted","Data":"e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f"}
Oct 30 23:40:27.659387 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:27.659078    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" event={"ID":"70538c08-5ce2-41fa-ac84-d6427beb113b","Type":"ContainerStarted","Data":"3d0dbb8499d9b0818d4aaab2ad174e3e11f121a37c6cc2747eec7780051058dc"}
Oct 30 23:40:27.728396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:27.724136    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podStartSLOduration=2.7241144889999998 podStartE2EDuration="2.724114489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:27.709759645 +0000 UTC m=+1286.636397228" watchObservedRunningTime="2026-10-30 23:40:27.724114489 +0000 UTC m=+1286.650752072"
Oct 30 23:40:27.907004 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:27.906915    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:40:27.907266 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:27.907235    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:40:27.955451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:27.955415    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 23:40:27.955774 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:27.955727    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 23:40:28.238461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:28.235667    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:28.238461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:28.235834    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:40:28.238461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:28.235878    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 23:40:28.906398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:28.905828    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:40:28.906398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:28.905893    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:40:28.953288 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:28.952782    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 23:40:28.953288 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:28.952845    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 23:40:29.086410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:29.083589    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:29.152397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:29.149744    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:29.236790 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:29.236689    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:40:29.236975 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:29.236782    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 23:40:29.903390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:29.902853    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:40:29.903390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:29.902920    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:40:29.955919 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:29.955881    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 23:40:29.956555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:29.956524    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 23:40:30.259454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:30.258529    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:40:30.259454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:30.258614    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 23:40:30.337397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:30.333787    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:40:30.606629 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:30.605852    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:40:30.905455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:30.901819    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:40:30.905455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:30.901897    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:40:30.957683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:30.956534    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 23:40:30.957683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:30.956629    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 23:40:31.249504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:31.244121    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:40:31.249504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:31.244184    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 23:40:31.938394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:31.937591    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:40:31.938394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:31.937667    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:40:31.953390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:31.953335    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 23:40:31.953703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:31.953674    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 23:40:32.244458 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:32.243650    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:40:32.244458 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:32.243713    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 23:40:32.901910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:32.901875    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:40:32.902237 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:32.902192    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:40:32.951070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:32.951036    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 23:40:32.951252 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:32.951094    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 23:40:33.233425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:33.233340    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:40:33.233578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:33.233425    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 23:40:34.057993 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:34.057960    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:34.059532 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:34.059482    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:34.060607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:34.060585    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 30 23:40:34.199010 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:34.198981    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:34.200614 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:34.200594    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:34.204064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:34.204026    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 30 23:40:34.234897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:34.234863    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:40:34.235222 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:34.235186    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 23:40:35.237999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:35.237969    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:35.238507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:35.238494    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:35.239355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:35.239335    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:40:35.648860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:35.648825    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:40:36.056657 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:36.056623    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:36.201239 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:36.201193    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:40.982268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:40.982218    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"]
Oct 30 23:40:40.988499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:40.988474    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:40.999146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:40.999121    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"]
Oct 30 23:40:41.152753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.152718    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2043db-3020-4355-b7d0-49f148914780-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.152953 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.152834    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.152953 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.152917    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj5lb\" (UniqueName: \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-kube-api-access-dj5lb\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.153090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.153009    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.153090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.153070    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.153167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.153143    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.153205 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.153185    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2043db-3020-4355-b7d0-49f148914780-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.153241 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.153204    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.153338 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.153311    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.254513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.254423    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.254513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.254510    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj5lb\" (UniqueName: \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-kube-api-access-dj5lb\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.254745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.254549    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.254745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.254604    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.254745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.254640    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.254745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.254682    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2043db-3020-4355-b7d0-49f148914780-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.254745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.254710    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.254745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.254739    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.255040 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.254793    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2043db-3020-4355-b7d0-49f148914780-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.255040 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.254821    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.255148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.255100    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.255213 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.255184    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.255275 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.255237    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.255626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.255604    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2043db-3020-4355-b7d0-49f148914780-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.257096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.257077    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.257418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.257362    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2043db-3020-4355-b7d0-49f148914780-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.267061 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.267034    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj5lb\" (UniqueName: \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-kube-api-access-dj5lb\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.267426 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.267405    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.317201 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.317121    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:41.612627 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.612595    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"]
Oct 30 23:40:41.614110 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:40:41.614086    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2043db_3020_4355_b7d0_49f148914780.slice/crio-da2ca85ac1668f87d991dd80cc3410d818ae62103315a2e4935e76afdc25fbec WatchSource:0}: Error finding container da2ca85ac1668f87d991dd80cc3410d818ae62103315a2e4935e76afdc25fbec: Status 404 returned error can't find the container with id da2ca85ac1668f87d991dd80cc3410d818ae62103315a2e4935e76afdc25fbec
Oct 30 23:40:41.616841 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.616806    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:41.616945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.616877    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:41.616945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.616908    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:41.937496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.937408    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" event={"ID":"ca2043db-3020-4355-b7d0-49f148914780","Type":"ContainerStarted","Data":"6c119b261e59d8273f86b0f31f75845ebda9eb5b2a684a7eba63cf968dd43ed2"}
Oct 30 23:40:41.937496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.937473    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" event={"ID":"ca2043db-3020-4355-b7d0-49f148914780","Type":"ContainerStarted","Data":"da2ca85ac1668f87d991dd80cc3410d818ae62103315a2e4935e76afdc25fbec"}
Oct 30 23:40:41.965188 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:41.965137    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podStartSLOduration=1.965123144 podStartE2EDuration="1.965123144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:41.96096603 +0000 UTC m=+1300.887603612" watchObservedRunningTime="2026-10-30 23:40:41.965123144 +0000 UTC m=+1300.891760724"
Oct 30 23:40:42.317921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:42.317884    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:42.319278 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:42.319256    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:40:42.319356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:42.319297    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:40:43.317830 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:43.317793    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:40:43.318033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:43.317858    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:40:44.317572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:44.317530    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:40:44.317756 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:44.317600    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:40:45.318119 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:45.318083    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:40:45.318551 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:45.318141    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:40:46.142136 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:46.142099    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:46.317664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:46.317621    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:40:46.317844 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:46.317690    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:40:47.318167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:47.318134    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:40:47.318594 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:47.318187    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:40:48.318272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:48.318228    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:40:48.318715 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:48.318300    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:40:49.324142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:49.324105    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:49.324664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:49.324535    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:49.325709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:49.325686    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:40:51.203287 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:51.203257    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:54.598696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.598644    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"]
Oct 30 23:40:54.614532 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.614042    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"]
Oct 30 23:40:54.614532 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.614162    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.626542 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.626519    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ef03a\"/\"distinct-w2-9ef03a-kserve-self-signed-certs\""
Oct 30 23:40:54.749396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.745825    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-model-cache\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.749396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.745923    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tls-certs\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.749396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.746033    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-home\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.749396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.746110    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5w5v\" (UniqueName: \"kubernetes.io/projected/16bfafce-6391-4dc5-8eb4-2a77f0f22164-kube-api-access-c5w5v\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.749396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.746176    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-dshm\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.749396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.746198    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tmp-dir\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.847577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.847536    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-home\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.847784 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.847606    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5w5v\" (UniqueName: \"kubernetes.io/projected/16bfafce-6391-4dc5-8eb4-2a77f0f22164-kube-api-access-c5w5v\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.847784 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.847662    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-dshm\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.847784 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.847684    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tmp-dir\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.847784 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.847741    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-model-cache\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.848031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.847803    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tls-certs\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.848085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.848032    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-home\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.848296 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.848269    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tmp-dir\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.848857 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.848781    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-model-cache\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.851164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.851137    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-dshm\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.852912 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.852886    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tls-certs\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.861115 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.861082    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5w5v\" (UniqueName: \"kubernetes.io/projected/16bfafce-6391-4dc5-8eb4-2a77f0f22164-kube-api-access-c5w5v\") pod \"distinct-w2-9ef03a-kserve-6b898876c-m69sm\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") " pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:54.935233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:54.935200    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:55.275329 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.275299    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"]
Oct 30 23:40:55.765108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.765016    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"]
Oct 30 23:40:55.769135 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.769112    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.783895 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.783865    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"]
Oct 30 23:40:55.842547 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:55.842511    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:55.842722 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:55.842522    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:55.866770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.866731    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.866916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.866806    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.866916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.866897    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.867027 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.866968    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.867167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.867145    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j4b8\" (UniqueName: \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-kube-api-access-2j4b8\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.867260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.867241    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.867318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.867302    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.867387 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.867352    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.867451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.867396    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.968994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.968946    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4b8\" (UniqueName: \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-kube-api-access-2j4b8\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969029    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969072    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969107    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969132    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969201    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969254    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969281    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969343    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969717 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969690    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969891 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969864    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.969993 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.969953    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.970270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.970243    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.970447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.970298    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.971926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.971905    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.972716 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.972697    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.982365 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.982324    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:55.982700 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:55.982675    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4b8\" (UniqueName: \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-kube-api-access-2j4b8\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:56.061602 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.061550    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"]
Oct 30 23:40:56.070527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.070493    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" event={"ID":"16bfafce-6391-4dc5-8eb4-2a77f0f22164","Type":"ContainerStarted","Data":"9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344"}
Oct 30 23:40:56.070527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.070530    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" event={"ID":"16bfafce-6391-4dc5-8eb4-2a77f0f22164","Type":"ContainerStarted","Data":"71acea1e4045fb8c0b88dad0807545b59f34c4be51dc8e368586dae350dbe7bd"}
Oct 30 23:40:56.070769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.070583    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.080831 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.080795    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"]
Oct 30 23:40:56.091185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.091153    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:56.108910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.106604    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"]
Oct 30 23:40:56.116605 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.116558    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podStartSLOduration=2.116542505 podStartE2EDuration="2.116542505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:56.111822278 +0000 UTC m=+1315.038459859" watchObservedRunningTime="2026-10-30 23:40:56.116542505 +0000 UTC m=+1315.043180087"
Oct 30 23:40:56.118031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.118007    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.129432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.125944    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"]
Oct 30 23:40:56.157075 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:56.157044    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f8605_1c38_4d02_9ebd_1c4101193dbc.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:40:56.171429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171281    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.171429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171326    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.171429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171419    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.171718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171511    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.171718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171542    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.171718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171573    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49e8515-359e-483c-81aa-51c63c2e37e3-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.171718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171615    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.171718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171639    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.171718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171683    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.171718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171714    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h46l\" (UniqueName: \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-kube-api-access-6h46l\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.172067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171791    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.172067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171829    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.172067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171863    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.172067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171897    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.172067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171939    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqd9\" (UniqueName: \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-kube-api-access-mcqd9\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.172067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.171987    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.172067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.172039    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.172452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.172145    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.274003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.273958    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.274214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274026    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.274214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274060    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49e8515-359e-483c-81aa-51c63c2e37e3-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.274214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274112    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.274214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274137    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.274214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274183    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.274214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274212    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h46l\" (UniqueName: \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-kube-api-access-6h46l\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.274603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274276    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.274603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274314    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.274603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274359    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.274603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274432    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.274603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274466    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqd9\" (UniqueName: \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-kube-api-access-mcqd9\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.274603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274512    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.274603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274560    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.274971 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274661    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.274971 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274725    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.274971 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274748    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.274971 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.274788    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.275555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.275531    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.276464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.276401    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-certs\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.277542 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.277290    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.278405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.277931    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.278405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.278198    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.278585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.278453    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-credential-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.278762 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.278737    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.279101 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.279063    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49e8515-359e-483c-81aa-51c63c2e37e3-istiod-ca-cert\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.279352 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.279308    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-socket\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.279642 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.279608    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.280085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.280056    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-data\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.280865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.280842    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.283349 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.283326    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-envoy\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.285695 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.285671    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-podinfo\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.323535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.314560    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.323535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.318932    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-token\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.323535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.319478    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h46l\" (UniqueName: \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-kube-api-access-6h46l\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.327213 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.327183    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqd9\" (UniqueName: \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-kube-api-access-mcqd9\") pod \"e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") " pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.417808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.417766    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:56.422365 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.422343    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"]
Oct 30 23:40:56.423762 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:40:56.423735    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473d73e9_7ff3_4f29_b8c6_5d40e71224af.slice/crio-8f4aa5f20bb1be05673322881c2766ae46c5f9f5b81d51c647aeb95926e912dc WatchSource:0}: Error finding container 8f4aa5f20bb1be05673322881c2766ae46c5f9f5b81d51c647aeb95926e912dc: Status 404 returned error can't find the container with id 8f4aa5f20bb1be05673322881c2766ae46c5f9f5b81d51c647aeb95926e912dc
Oct 30 23:40:56.429512 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.429472    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:56.429595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.429562    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:56.429630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.429606    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:56.462085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.462049    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:56.726558 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:40:56.726529    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9d3e62_ce88_426b_a0ba_9d41d1ddbaf5.slice/crio-78561450a27d68eb3281b77d7800089c00ee89b08fe408bf36c42938ab0dc9af WatchSource:0}: Error finding container 78561450a27d68eb3281b77d7800089c00ee89b08fe408bf36c42938ab0dc9af: Status 404 returned error can't find the container with id 78561450a27d68eb3281b77d7800089c00ee89b08fe408bf36c42938ab0dc9af
Oct 30 23:40:56.727106 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.727085    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"]
Oct 30 23:40:56.729914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.729878    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:56.730030 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.729955    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:56.730030 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.729986    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:56.802607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.802313    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"]
Oct 30 23:40:56.803133 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:40:56.803105    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49e8515_359e_483c_81aa_51c63c2e37e3.slice/crio-3c2b6388eb0285c58fb57af43e8741a1677f790f6cc01f96dc875aff9c5140b7 WatchSource:0}: Error finding container 3c2b6388eb0285c58fb57af43e8741a1677f790f6cc01f96dc875aff9c5140b7: Status 404 returned error can't find the container with id 3c2b6388eb0285c58fb57af43e8741a1677f790f6cc01f96dc875aff9c5140b7
Oct 30 23:40:56.806562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.806517    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:56.806675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.806608    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:56.806675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.806650    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:40:56.936302 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:56.935902    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:40:57.084144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.084101    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" event={"ID":"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5","Type":"ContainerStarted","Data":"4cbe4578003fe9a98b72fca36b51f02ef1a331594fb38781a44ca0e9ab8a7c0c"}
Oct 30 23:40:57.084362 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.084165    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" event={"ID":"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5","Type":"ContainerStarted","Data":"78561450a27d68eb3281b77d7800089c00ee89b08fe408bf36c42938ab0dc9af"}
Oct 30 23:40:57.087994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.087360    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" event={"ID":"e49e8515-359e-483c-81aa-51c63c2e37e3","Type":"ContainerStarted","Data":"8637634dcced5385de37619eb15716d48de80ff0b0059ed80d03ac1673e10f7b"}
Oct 30 23:40:57.087994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.087464    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" event={"ID":"e49e8515-359e-483c-81aa-51c63c2e37e3","Type":"ContainerStarted","Data":"3c2b6388eb0285c58fb57af43e8741a1677f790f6cc01f96dc875aff9c5140b7"}
Oct 30 23:40:57.089918 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.089886    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" event={"ID":"473d73e9-7ff3-4f29-b8c6-5d40e71224af","Type":"ContainerStarted","Data":"a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d"}
Oct 30 23:40:57.090056 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.089928    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" event={"ID":"473d73e9-7ff3-4f29-b8c6-5d40e71224af","Type":"ContainerStarted","Data":"8f4aa5f20bb1be05673322881c2766ae46c5f9f5b81d51c647aeb95926e912dc"}
Oct 30 23:40:57.091945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.091903    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:40:57.093125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.093105    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:40:57.093229 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.093155    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:40:57.112204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.112162    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podStartSLOduration=1.112151208 podStartE2EDuration="1.112151208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:57.109638425 +0000 UTC m=+1316.036276006" watchObservedRunningTime="2026-10-30 23:40:57.112151208 +0000 UTC m=+1316.038788789"
Oct 30 23:40:57.142114 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.142062    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podStartSLOduration=2.142048044 podStartE2EDuration="2.142048044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:57.139571049 +0000 UTC m=+1316.066208630" watchObservedRunningTime="2026-10-30 23:40:57.142048044 +0000 UTC m=+1316.068685753"
Oct 30 23:40:57.172730 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.172672    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podStartSLOduration=1.172658216 podStartE2EDuration="1.172658216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:57.170054976 +0000 UTC m=+1316.096692559" watchObservedRunningTime="2026-10-30 23:40:57.172658216 +0000 UTC m=+1316.099295797"
Oct 30 23:40:57.418184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.418104    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:40:57.420126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.420094    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:40:57.420239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.420156    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:40:57.446594 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:40:57.446563    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:40:57.483404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.482956    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:40:57.494422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.491511    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:40:57.494422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:57.491577    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:40:58.094974 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:58.094938    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:40:58.099644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:58.099606    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:40:58.429453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:58.421467    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:40:58.429453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:58.421528    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:40:58.466397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:58.465799    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:40:58.466397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:58.465873    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:40:59.091821 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:59.091784    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:40:59.092117 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:59.092069    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:40:59.420853 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:59.420764    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:40:59.421479 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:59.421447    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:40:59.462696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:59.462662    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:40:59.462973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:40:59.462946    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:41:00.092004 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:00.091953    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:41:00.092346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:00.092293    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:41:00.421565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:00.421479    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:41:00.422131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:00.422101    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:41:00.507407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:00.483016    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:41:00.507407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:00.483080    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:41:01.093395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:01.092760    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:41:01.093395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:01.092820    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:41:01.420457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:01.420091    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:41:01.420457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:01.420147    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:41:01.463396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:01.462911    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:41:01.463396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:01.462968    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:41:01.489507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:01.489462    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:01.711282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:01.710260    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:01.970579 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:01.970472    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 30 23:41:02.091836 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:02.091799    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:41:02.092114 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:02.092085    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:41:02.283406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:02.281427    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:02.383398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:02.380694    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:02.421407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:02.419748    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:41:02.421407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:02.419807    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:41:02.466409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:02.464736    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:41:02.466409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:02.464894    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:41:02.616592 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:02.616492    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 23:41:03.123840 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.123808    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:41:03.139545 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.139498    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:41:03.295398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.291538    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:41:03.394460 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.383858    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:03.441619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.441582    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:41:03.441881 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.441855    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:41:03.479619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.465342    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:41:03.480875 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.480839    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:41:03.723449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.551009    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:41:03.723449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.635502    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:41:03.723449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.589769    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:41:03.723449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.617643    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:03.723449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.613420    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:41:03.723449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.617681    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:03.723449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.640957    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:41:03.767511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.767468    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:41:03.823405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.812522    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"]
Oct 30 23:41:03.956711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.881747    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:03.956711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.921313    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:03.956711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.946569    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:03.956711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.946797    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe60fd8-3fa6-46ec-9447-f122274a818c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:03.956711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.946862    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:03.956711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.947044    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:03.956711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.947159    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:03.956711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.947193    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqtbx\" (UniqueName: \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-kube-api-access-sqtbx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:03.991075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.967482    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 23:41:03.991679 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.991518    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 23:41:04.000592 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:03.999878    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qbh58\""
Oct 30 23:41:04.081907 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.081869    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.082150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.082133    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.082316 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.082300    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqtbx\" (UniqueName: \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-kube-api-access-sqtbx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.082508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.082494    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.082727 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.082713    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.083011 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.082997    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe60fd8-3fa6-46ec-9447-f122274a818c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.083314 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.083143    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.142396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.142344    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:41:04.145326 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.145295    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:41:04.157519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.143739    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.158908 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.158882    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"]
Oct 30 23:41:04.181408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.174979    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.213602 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.213569    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 23:41:04.307105 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.302591    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:04.309965 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.309279    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.321902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.321061    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqtbx\" (UniqueName: \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-kube-api-access-sqtbx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.321902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.321858    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.401394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.399747    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 23:41:04.468447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.465225    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:04.468447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.465299    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:41:04.468447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.465362    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:41:04.468447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.465488    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:41:04.468447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.465518    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:41:04.536273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.509121    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe60fd8-3fa6-46ec-9447-f122274a818c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.653578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.653440    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:04.654162 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.654128    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:04.670393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.654588    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:04.671977 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.671944    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:41:04.672115 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.672022    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:41:04.672182 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.672151    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dnqr9\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.763601 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.763572    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:04.768266 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.768230    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:04.768906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.768884    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:41:04.949318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:04.949291    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:41:05.095235 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.094776    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:41:05.095235 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.094835    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:41:05.314531 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.314482    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:05.316148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.315688    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:05.316148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.315738    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:05.362626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.362484    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:05.362626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.362548    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:05.375605 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.375547    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:05.375813 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.375614    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:05.404828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.404771    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:05.419086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.419049    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:41:05.419351 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.419326    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:41:05.462906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.462870    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:41:05.463210 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.463169    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:41:05.468202 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.468128    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:05.468869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.468849    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:41:05.679562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.678538    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:05.679562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.678925    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:05.723405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.717430    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:41:05.734864 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.734530    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:05.734864 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.734587    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:05.796417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.796328    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:05.889818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.889780    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:41:05.891702 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:05.891678    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:41:06.221541 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.221504    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:41:06.252072 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.222357    4555 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Oct 30 23:41:06.252072 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.252010    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:41:06.319744 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.319696    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:06.391546 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.391496    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:06.391842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.391811    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:06.423832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.423799    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:41:06.424138 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.424094    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:41:06.444194 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.442071    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 23:41:06.506447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.473412    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:06.616397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.588498    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:41:06.616397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.588564    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:41:06.668015 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.667980    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"]
Oct 30 23:41:06.756409 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:41:06.732452    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe60fd8_3fa6_46ec_9447_f122274a818c.slice/crio-ce9d00ee54ecdfa47aedc544944c4373884cfd74ef80d24883a37c4af8939702 WatchSource:0}: Error finding container ce9d00ee54ecdfa47aedc544944c4373884cfd74ef80d24883a37c4af8939702: Status 404 returned error can't find the container with id ce9d00ee54ecdfa47aedc544944c4373884cfd74ef80d24883a37c4af8939702
Oct 30 23:41:06.756409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.748068    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:06.826865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.826013    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:06.927399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:06.911161    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 23:41:07.147555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.147526    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:41:07.155436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.147805    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:41:07.335220 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.335165    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:07.448981 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.448881    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:41:07.448981 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.448969    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:41:07.490311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.490261    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:41:07.525588 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.525551    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9" event={"ID":"4fe60fd8-3fa6-46ec-9447-f122274a818c","Type":"ContainerStarted","Data":"ce9d00ee54ecdfa47aedc544944c4373884cfd74ef80d24883a37c4af8939702"}
Oct 30 23:41:07.563032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.508144    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:41:07.563393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.563336    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:41:07.564055 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.542512    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:07.642440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.637095    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"]
Oct 30 23:41:07.699448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.689568    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:41:07.699448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.689607    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"]
Oct 30 23:41:07.699448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.690401    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:07.699448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.691001    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:07.699448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.698647    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:41:07.699448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.698807    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:41:07.752662 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.752636    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:41:07.753050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.753033    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:41:07.792399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.758875    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:41:07.800034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.800003    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-423201fe-cn7hs\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:07.800394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.800268    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdrb\" (UniqueName: \"kubernetes.io/projected/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-kube-api-access-2jdrb\") pod \"maas-api-revoke-sub-423201fe-cn7hs\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:07.825309 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.825274    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-423201fe-cn7hs\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:07.931237 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.931196    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-423201fe-cn7hs\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:07.932131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.932106    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-423201fe-cn7hs\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:07.933312 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.933279    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdrb\" (UniqueName: \"kubernetes.io/projected/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-kube-api-access-2jdrb\") pod \"maas-api-revoke-sub-423201fe-cn7hs\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:07.934218 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.934192    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-423201fe-cn7hs\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:07.938441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.938418    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-423201fe-cn7hs\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:07.951676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:07.951603    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdrb\" (UniqueName: \"kubernetes.io/projected/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-kube-api-access-2jdrb\") pod \"maas-api-revoke-sub-423201fe-cn7hs\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:08.093053 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:08.093014    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:41:08.093237 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:08.093075    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:41:08.127491 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:08.127456    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:08.418546 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:08.418507    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 23:41:08.418748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:08.418577    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 23:41:08.463317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:08.463273    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:41:08.463540 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:08.463356    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:41:08.504120 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:08.504089    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"]
Oct 30 23:41:09.257445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:09.255110    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:41:09.257445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:09.256050    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:41:09.258797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:09.258772    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:41:09.455692 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:09.455346    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs" event={"ID":"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90","Type":"ContainerStarted","Data":"a654c43afec7524c57608022129406482a7c95fafebc5d8e6430f46dbae65436"}
Oct 30 23:41:09.455692 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:09.455431    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs" event={"ID":"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90","Type":"ContainerStarted","Data":"28948ce65b94f8e3b1eb2bf2c483becc680e5fdc61a4009d1c6ac40f1c126aa3"}
Oct 30 23:41:10.473394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:10.472495    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:10.473394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:10.472553    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:10.477859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:10.477472    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:10.477859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:10.477538    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:10.486360 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:10.485211    4555 generic.go:356] "Generic (PLEG): container finished" podID="d8b1517b-3351-4bc4-bd8f-d8dd19e32e90" containerID="a654c43afec7524c57608022129406482a7c95fafebc5d8e6430f46dbae65436" exitCode=0
Oct 30 23:41:10.486360 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:10.486288    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs" event={"ID":"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90","Type":"ContainerDied","Data":"a654c43afec7524c57608022129406482a7c95fafebc5d8e6430f46dbae65436"}
Oct 30 23:41:11.366693 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.366658    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:41:11.368284 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.368136    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:41:11.369095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.369073    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:41:11.486391 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.467113    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:41:11.521644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.521430    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:11.521644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.521487    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:11.822347 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.822314    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:11.847582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.847542    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:41:11.849144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.849122    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:41:11.879217 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.877719    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-openshift-ca\") pod \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") "
Oct 30 23:41:11.879217 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.878030    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-kube-api-access-2jdrb\" (UniqueName: \"kubernetes.io/projected/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-kube-api-access-2jdrb\") pod \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") "
Oct 30 23:41:11.879217 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.878067    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-xks-ca\") pod \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\" (UID: \"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90\") "
Oct 30 23:41:11.880081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.879763    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-openshift-ca" pod "d8b1517b-3351-4bc4-bd8f-d8dd19e32e90" (UID: "d8b1517b-3351-4bc4-bd8f-d8dd19e32e90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:41:11.893569 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.892689    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-kube-api-access-2jdrb" pod "d8b1517b-3351-4bc4-bd8f-d8dd19e32e90" (UID: "d8b1517b-3351-4bc4-bd8f-d8dd19e32e90"). InnerVolumeSpecName "kube-api-access-2jdrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:41:11.893569 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.893494    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-xks-ca" pod "d8b1517b-3351-4bc4-bd8f-d8dd19e32e90" (UID: "d8b1517b-3351-4bc4-bd8f-d8dd19e32e90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:41:11.980306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.980227    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdrb\" (UniqueName: \"kubernetes.io/projected/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-kube-api-access-2jdrb\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:11.980306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.980264    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:11.980306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:11.980279    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:12.508662 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:12.508629    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"
Oct 30 23:41:12.513167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:12.512421    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs" event={"ID":"d8b1517b-3351-4bc4-bd8f-d8dd19e32e90","Type":"ContainerDied","Data":"28948ce65b94f8e3b1eb2bf2c483becc680e5fdc61a4009d1c6ac40f1c126aa3"}
Oct 30 23:41:12.513167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:12.512475    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28948ce65b94f8e3b1eb2bf2c483becc680e5fdc61a4009d1c6ac40f1c126aa3"
Oct 30 23:41:13.482491 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:13.482457    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:13.482630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:13.482522    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:14.530576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:14.530535    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9" event={"ID":"4fe60fd8-3fa6-46ec-9447-f122274a818c","Type":"ContainerStarted","Data":"ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa"}
Oct 30 23:41:14.531075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:14.530598    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:14.532733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:14.532706    4555 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dnqr9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:41:14.532881 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:14.532778    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9" podUID="4fe60fd8-3fa6-46ec-9447-f122274a818c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:41:14.557122 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:14.557067    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9" podStartSLOduration=5.405743634 podStartE2EDuration="11.557050727s" totalImagesPullingTime="6.151307093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:14.554690195 +0000 UTC m=+1333.481327766" watchObservedRunningTime="2026-10-30 23:41:14.557050727 +0000 UTC m=+1333.483688289"
Oct 30 23:41:14.953741 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:14.953644    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:41:14.970635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:14.970605    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:41:15.537761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:15.537736    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 30 23:41:25.694401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.694350    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"]
Oct 30 23:41:25.695102 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.694988    4555 cpu_manager.go:414] "containerMap: removing container" podUID="d8b1517b-3351-4bc4-bd8f-d8dd19e32e90" containerName="revoke-keys"
Oct 30 23:41:25.695102 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.695009    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b1517b-3351-4bc4-bd8f-d8dd19e32e90" containerName="revoke-keys"
Oct 30 23:41:25.695307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.695148    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b1517b-3351-4bc4-bd8f-d8dd19e32e90" containerName="revoke-keys"
Oct 30 23:41:25.699533 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.699511    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.722941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.722910    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"]
Oct 30 23:41:25.780625 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.780575    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.780819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.780725    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9261021-433e-4641-927b-1893dd0d1f24-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.780819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.780786    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.780938 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.780896    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.781002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.780948    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.781059 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.781039    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9261021-433e-4641-927b-1893dd0d1f24-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.781123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.781106    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.781200 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.781184    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8ql\" (UniqueName: \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-kube-api-access-nx8ql\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.781312 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.781296    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882179 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882127    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882389 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882201    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882389 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882293    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9261021-433e-4641-927b-1893dd0d1f24-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882389 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882357    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882419    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882462    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882517    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9261021-433e-4641-927b-1893dd0d1f24-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882554    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882611    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882639    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8ql\" (UniqueName: \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-kube-api-access-nx8ql\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.882798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.882660    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.883132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.883105    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.883429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.883406    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.883587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.883560    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9261021-433e-4641-927b-1893dd0d1f24-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.885041 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.885020    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.885433 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.885410    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9261021-433e-4641-927b-1893dd0d1f24-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.896842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.896817    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.897035 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.896941    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8ql\" (UniqueName: \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-kube-api-access-nx8ql\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:25.993543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:25.993432    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"]
Oct 30 23:41:26.009090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.008812    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.016477 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.016449    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"]
Oct 30 23:41:26.032201 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.032179    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:26.068964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.068916    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"]
Oct 30 23:41:26.074559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.074534    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.119978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.119946    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"]
Oct 30 23:41:26.207072 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207034    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.207271 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207092    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.207271 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207146    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-496df\" (UniqueName: \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-kube-api-access-496df\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.207440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207273    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.207440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207317    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.207440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207359    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.207440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207402    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.207663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207471    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.207663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207602    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.207663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207627    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.207663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207643    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.207879 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207669    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.207879 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207708    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.207879 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207792    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.207879 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207828    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.208099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207888    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.208099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.207954    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj94z\" (UniqueName: \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-kube-api-access-gj94z\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.208099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.208018    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.309958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.309921    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.310141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.309970    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.310141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310015    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj94z\" (UniqueName: \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-kube-api-access-gj94z\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.310141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310105    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.310319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310166    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.310319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310226    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.310319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310249    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-496df\" (UniqueName: \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-kube-api-access-496df\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.310319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310277    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.310319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310304    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.310618 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310452    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-certs\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.310618 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310450    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.310618 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310496    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.310618 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310525    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.310860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310632    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.310860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310673    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.310860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310700    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.310860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310741    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.310860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310768    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.310860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310799    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.311190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310856    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-credential-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.311190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.310899    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-data\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.311958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.311247    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-socket\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.311958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.311416    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-data\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.311958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.311688    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.311958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.311739    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-certs\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.311958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.311782    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istiod-ca-cert\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.312301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.311994    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istiod-ca-cert\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.312301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.312055    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-credential-socket\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.313139 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.313118    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-envoy\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.313645 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.313615    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-podinfo\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.313947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.313932    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-podinfo\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.314122 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.314098    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-envoy\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.352360 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.352335    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj94z\" (UniqueName: \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-kube-api-access-gj94z\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.364109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.364082    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-token\") pod \"e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") " pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.364225 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.364096    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-token\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.374835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.374811    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-496df\" (UniqueName: \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-kube-api-access-496df\") pod \"e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") " pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.420312 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.420130    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:26.422360 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:41:26.422331    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9261021_433e_4641_927b_1893dd0d1f24.slice/crio-9d10673592d4f440a00e37c82536f4a2e99bdc773f2f0ac9c93d4762e369f674 WatchSource:0}: Error finding container 9d10673592d4f440a00e37c82536f4a2e99bdc773f2f0ac9c93d4762e369f674: Status 404 returned error can't find the container with id 9d10673592d4f440a00e37c82536f4a2e99bdc773f2f0ac9c93d4762e369f674
Oct 30 23:41:26.422521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.422442    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"]
Oct 30 23:41:26.425298 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.425262    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:26.425361 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.425344    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:26.425437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.425410    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:26.628810 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.628716    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:26.634288 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.634257    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" event={"ID":"f9261021-433e-4641-927b-1893dd0d1f24","Type":"ContainerStarted","Data":"aafd2075a3546fc21990cd7846023d48f4782c9384a3f11262689bc69cbf90ef"}
Oct 30 23:41:26.634425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.634325    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" event={"ID":"f9261021-433e-4641-927b-1893dd0d1f24","Type":"ContainerStarted","Data":"9d10673592d4f440a00e37c82536f4a2e99bdc773f2f0ac9c93d4762e369f674"}
Oct 30 23:41:26.720860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.720661    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podStartSLOduration=1.720640081 podStartE2EDuration="1.720640081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:26.681429816 +0000 UTC m=+1345.608067421" watchObservedRunningTime="2026-10-30 23:41:26.720640081 +0000 UTC m=+1345.647277663"
Oct 30 23:41:26.750009 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.749970    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"]
Oct 30 23:41:26.884808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.884766    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:26.885031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.884856    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:26.885031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.884901    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:26.943670 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:41:26.943643    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65b666d_2a91_451a_9da2_3cdf0a99be7a.slice/crio-540cb91af2cfff53e63b6a136df574acb45f556575dfe3b2348072da091edc56 WatchSource:0}: Error finding container 540cb91af2cfff53e63b6a136df574acb45f556575dfe3b2348072da091edc56: Status 404 returned error can't find the container with id 540cb91af2cfff53e63b6a136df574acb45f556575dfe3b2348072da091edc56
Oct 30 23:41:26.946406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.946327    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:26.946490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.946458    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:26.946554 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.946537    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:26.949362 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:26.949341    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"]
Oct 30 23:41:27.033483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:27.033455    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:27.035004 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:27.034827    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:27.035004 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:27.034886    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:27.454469 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:41:27.454429    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:41:27.657435 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:27.656599    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" event={"ID":"e65b666d-2a91-451a-9da2-3cdf0a99be7a","Type":"ContainerStarted","Data":"49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5"}
Oct 30 23:41:27.657435 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:27.656676    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" event={"ID":"e65b666d-2a91-451a-9da2-3cdf0a99be7a","Type":"ContainerStarted","Data":"540cb91af2cfff53e63b6a136df574acb45f556575dfe3b2348072da091edc56"}
Oct 30 23:41:27.663708 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:27.660401    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" event={"ID":"c907bcbf-31c9-472f-97c2-58b87c67e7f1","Type":"ContainerStarted","Data":"0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da"}
Oct 30 23:41:27.663708 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:27.660453    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" event={"ID":"c907bcbf-31c9-472f-97c2-58b87c67e7f1","Type":"ContainerStarted","Data":"fa7c9a1c6c39649eb24fe81632796d2903b3ffecc5aad7cccee6aa9443cfc3ea"}
Oct 30 23:41:27.731451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:27.730683    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podStartSLOduration=2.730660968 podStartE2EDuration="2.730660968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:27.706842858 +0000 UTC m=+1346.633480440" watchObservedRunningTime="2026-10-30 23:41:27.730660968 +0000 UTC m=+1346.657298550"
Oct 30 23:41:28.037477 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:28.036187    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:28.037477 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:28.036249    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:28.433624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:28.433537    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:28.433942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:28.433919    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:41:28.434095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:28.434073    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 23:41:28.652399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:28.645742    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:28.652399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:28.650966    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:41:28.652399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:28.651022    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 23:41:29.039449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:29.037552    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:29.039449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:29.037611    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:29.351219 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:29.351105    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podStartSLOduration=3.351084389 podStartE2EDuration="3.351084389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:27.746918926 +0000 UTC m=+1346.673556508" watchObservedRunningTime="2026-10-30 23:41:29.351084389 +0000 UTC m=+1348.277721970"
Oct 30 23:41:29.352398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:29.352346    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:41:29.421528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:29.421474    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:41:29.421811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:29.421779    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 23:41:29.630918 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:29.630820    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:41:29.631182 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:29.631151    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 23:41:30.048797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:30.048765    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:30.049356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:30.049326    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:30.421885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:30.421791    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:41:30.422140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:30.422111    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 23:41:30.636395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:30.636013    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:41:30.636395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:30.636077    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 23:41:31.040764 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:31.040727    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:31.041063 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:31.041031    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:31.421894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:31.421804    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:41:31.422451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:31.422424    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 23:41:31.577398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:31.576578    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:31.629535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:31.629490    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:41:31.629826 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:31.629772    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 23:41:31.703270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:31.703148    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:31.725560 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:31.725508    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:31.757687 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:31.757647    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:32.047398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.038416    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:32.047398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.038478    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:32.324458 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.322475    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:41:32.441407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.438509    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:41:32.441407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.438576    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 23:41:32.597611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.597473    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:41:32.601403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.597937    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:41:32.638419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.635821    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:41:32.638419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.635882    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 23:41:32.683654 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.682481    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:32.845419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.838008    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:41:32.865527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.856495    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:41:32.943409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.938910    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:32.971425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:32.965000    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:33.039276 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.038475    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:33.039276 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.038548    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:33.323704 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.323659    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:33.425784 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.425749    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:41:33.426055 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.426026    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 23:41:33.535397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.526413    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:41:33.647475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.642031    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:33.657647 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.652471    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:33.691396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.683224    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:33.721410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.708446    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:41:33.814168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.813644    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:41:33.814168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.813705    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 23:41:33.890397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.878031    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:33.890397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.885687    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:33.957455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.955511    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 23:41:33.986191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:33.986158    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:41:34.036552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.011446    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:41:34.176352 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.125709    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:34.176662 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.176610    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:34.177288 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.177268    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:41:34.442489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.441956    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:41:34.442489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.442049    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 23:41:34.443074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.443043    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:34.445872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.445330    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:34.691398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.647857    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:41:34.715472 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.688598    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:41:34.823846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.823810    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 23:41:34.824481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.691622    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:41:34.825036 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.823481    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:34.825036 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.824964    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:34.945395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.887549    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:41:34.945395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.888246    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:41:34.945395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.889060    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:35.052455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:34.973639    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:35.201402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.054176    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:35.201402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.189410    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:35.201402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.180012    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:35.360817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.360725    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:35.361076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.361045    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:35.439350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.438890    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:41:35.439350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.438950    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 23:41:35.451563 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.451500    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:35.477397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.458460    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:35.488398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.487216    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:41:35.633886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.633797    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:41:35.634143 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.634112    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 23:41:35.670825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.670785    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:41:35.829404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.826523    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:41:35.884611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.884519    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:41:35.906401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.893350    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:41:35.963398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.922479    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:41:35.987037 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:35.986005    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:36.032890 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.032846    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:36.033194 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.033146    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:36.179073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.178972    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:41:36.231403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.208834    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 23:41:36.422508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.422470    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:41:36.431529 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.431441    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 23:41:36.458536 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.456951    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:36.460001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.459693    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:36.631397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.631097    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:41:36.631397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.631159    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 23:41:36.739446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.684464    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:36.739446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.686170    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:41:36.802435 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.801101    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:41:36.804495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.803977    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:41:36.968117 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:36.968089    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:41:37.033825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:37.033774    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 23:41:37.034087 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:37.033830    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 23:41:37.035756 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:37.035723    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:41:37.113835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:37.113788    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:41:37.224592 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:37.222140    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:41:37.421557 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:37.421464    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:41:37.421557 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:37.421513    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 23:41:37.630257 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:37.630216    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:41:37.630514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:37.630300    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 23:41:38.354934 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.354891    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"]
Oct 30 23:41:38.364293 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.363574    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.367635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.366714    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"]
Oct 30 23:41:38.373276 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.372460    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:41:38.373630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.373593    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:41:38.530569 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.530524    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b442ad33-sk8wb\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.535617 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.535578    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b442ad33-sk8wb\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.535896 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.535876    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxnr\" (UniqueName: \"kubernetes.io/projected/e62c9467-b445-44fb-b292-e554bfb80f51-kube-api-access-8rxnr\") pod \"maas-api-revoke-sub-b442ad33-sk8wb\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.661861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.637703    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b442ad33-sk8wb\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.661861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.637828    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxnr\" (UniqueName: \"kubernetes.io/projected/e62c9467-b445-44fb-b292-e554bfb80f51-kube-api-access-8rxnr\") pod \"maas-api-revoke-sub-b442ad33-sk8wb\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.661861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.637970    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b442ad33-sk8wb\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.661861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.638680    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b442ad33-sk8wb\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.661861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.641688    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b442ad33-sk8wb\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.661861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.656234    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxnr\" (UniqueName: \"kubernetes.io/projected/e62c9467-b445-44fb-b292-e554bfb80f51-kube-api-access-8rxnr\") pod \"maas-api-revoke-sub-b442ad33-sk8wb\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.698396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.692714    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:38.905445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.905416    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:38.907078 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.907058    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:38.919703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:38.919642    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:41:39.078184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:39.077593    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:41:39.078184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:39.077655    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:39.274394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:39.271638    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:39.274394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:39.272473    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:39.278394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:39.276780    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 30 23:41:39.330178 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:39.330152    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"]
Oct 30 23:41:39.535327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:39.535238    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:39.536409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:39.536390    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:39.540108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:39.540088    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 30 23:41:39.638622 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:39.638585    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb" event={"ID":"e62c9467-b445-44fb-b292-e554bfb80f51","Type":"ContainerStarted","Data":"f817b64664009cc03591c4590e480657d3ff110b67126bf2a5e61c6b56252ee8"}
Oct 30 23:41:40.647869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:40.647788    4555 generic.go:356] "Generic (PLEG): container finished" podID="e62c9467-b445-44fb-b292-e554bfb80f51" containerID="9bbe3738328d4227a080ee038a5706c2df6cb8defc6461f5d93ba8bbb4012506" exitCode=0
Oct 30 23:41:40.648212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:40.647925    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb" event={"ID":"e62c9467-b445-44fb-b292-e554bfb80f51","Type":"ContainerDied","Data":"9bbe3738328d4227a080ee038a5706c2df6cb8defc6461f5d93ba8bbb4012506"}
Oct 30 23:41:41.855432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:41.842834    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:41.959432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:41.959320    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-openshift-ca\") pod \"e62c9467-b445-44fb-b292-e554bfb80f51\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") "
Oct 30 23:41:41.959615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:41.959461    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-xks-ca\") pod \"e62c9467-b445-44fb-b292-e554bfb80f51\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") "
Oct 30 23:41:41.959615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:41.959593    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62c9467-b445-44fb-b292-e554bfb80f51-kube-api-access-8rxnr\" (UniqueName: \"kubernetes.io/projected/e62c9467-b445-44fb-b292-e554bfb80f51-kube-api-access-8rxnr\") pod \"e62c9467-b445-44fb-b292-e554bfb80f51\" (UID: \"e62c9467-b445-44fb-b292-e554bfb80f51\") "
Oct 30 23:41:41.960174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:41.959729    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-openshift-ca" pod "e62c9467-b445-44fb-b292-e554bfb80f51" (UID: "e62c9467-b445-44fb-b292-e554bfb80f51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:41:41.960174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:41.960144    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:41.967427 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:41.966744    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-xks-ca" pod "e62c9467-b445-44fb-b292-e554bfb80f51" (UID: "e62c9467-b445-44fb-b292-e554bfb80f51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:41:41.968046 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:41.968020    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62c9467-b445-44fb-b292-e554bfb80f51-kube-api-access-8rxnr" pod "e62c9467-b445-44fb-b292-e554bfb80f51" (UID: "e62c9467-b445-44fb-b292-e554bfb80f51"). InnerVolumeSpecName "kube-api-access-8rxnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:41:42.061746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.061714    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62c9467-b445-44fb-b292-e554bfb80f51-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:42.061746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.061743    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rxnr\" (UniqueName: \"kubernetes.io/projected/e62c9467-b445-44fb-b292-e554bfb80f51-kube-api-access-8rxnr\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:42.510850 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.510810    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"]
Oct 30 23:41:42.511691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.511671    4555 cpu_manager.go:414] "containerMap: removing container" podUID="e62c9467-b445-44fb-b292-e554bfb80f51" containerName="revoke-keys"
Oct 30 23:41:42.511767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.511694    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62c9467-b445-44fb-b292-e554bfb80f51" containerName="revoke-keys"
Oct 30 23:41:42.511885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.511873    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62c9467-b445-44fb-b292-e554bfb80f51" containerName="revoke-keys"
Oct 30 23:41:42.515926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.515901    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:42.522456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.522349    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"]
Oct 30 23:41:42.568385 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.568326    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6gsp\" (UniqueName: \"kubernetes.io/projected/93eae285-b8d0-4f49-b72a-d6de913834ee-kube-api-access-v6gsp\") pod \"maas-api-revoke-sub-5ada8fa5-kcklg\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:42.568574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.568455    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ada8fa5-kcklg\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:42.568717 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.568692    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ada8fa5-kcklg\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:42.663239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.663208    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"
Oct 30 23:41:42.663239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.663235    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb" event={"ID":"e62c9467-b445-44fb-b292-e554bfb80f51","Type":"ContainerDied","Data":"f817b64664009cc03591c4590e480657d3ff110b67126bf2a5e61c6b56252ee8"}
Oct 30 23:41:42.663505 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.663292    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f817b64664009cc03591c4590e480657d3ff110b67126bf2a5e61c6b56252ee8"
Oct 30 23:41:42.670248 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.670227    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6gsp\" (UniqueName: \"kubernetes.io/projected/93eae285-b8d0-4f49-b72a-d6de913834ee-kube-api-access-v6gsp\") pod \"maas-api-revoke-sub-5ada8fa5-kcklg\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:42.670361 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.670277    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ada8fa5-kcklg\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:42.670448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.670382    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ada8fa5-kcklg\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:42.671542 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.671274    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ada8fa5-kcklg\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:42.673361 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.673335    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ada8fa5-kcklg\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:42.683109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.683086    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6gsp\" (UniqueName: \"kubernetes.io/projected/93eae285-b8d0-4f49-b72a-d6de913834ee-kube-api-access-v6gsp\") pod \"maas-api-revoke-sub-5ada8fa5-kcklg\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:42.837347 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:42.836588    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:43.166000 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:41:43.165963    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93eae285_b8d0_4f49_b72a_d6de913834ee.slice/crio-23c81de3526ae2e4ddffe374a0265c413de6d80c15019c4cc1dbfdf178242bd3 WatchSource:0}: Error finding container 23c81de3526ae2e4ddffe374a0265c413de6d80c15019c4cc1dbfdf178242bd3: Status 404 returned error can't find the container with id 23c81de3526ae2e4ddffe374a0265c413de6d80c15019c4cc1dbfdf178242bd3
Oct 30 23:41:43.167659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:43.167460    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"]
Oct 30 23:41:43.672039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:43.671997    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg" event={"ID":"93eae285-b8d0-4f49-b72a-d6de913834ee","Type":"ContainerStarted","Data":"23c81de3526ae2e4ddffe374a0265c413de6d80c15019c4cc1dbfdf178242bd3"}
Oct 30 23:41:44.683161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:44.683070    4555 generic.go:356] "Generic (PLEG): container finished" podID="93eae285-b8d0-4f49-b72a-d6de913834ee" containerID="5f686030639209c05204e03a6b1dd024a2d86083b72ede640b8b6cf4200e56d5" exitCode=0
Oct 30 23:41:44.683624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:44.683163    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg" event={"ID":"93eae285-b8d0-4f49-b72a-d6de913834ee","Type":"ContainerDied","Data":"5f686030639209c05204e03a6b1dd024a2d86083b72ede640b8b6cf4200e56d5"}
Oct 30 23:41:45.864272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:45.864244    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:45.923906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:45.923875    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93eae285-b8d0-4f49-b72a-d6de913834ee-kube-api-access-v6gsp\" (UniqueName: \"kubernetes.io/projected/93eae285-b8d0-4f49-b72a-d6de913834ee-kube-api-access-v6gsp\") pod \"93eae285-b8d0-4f49-b72a-d6de913834ee\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") "
Oct 30 23:41:45.924086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:45.923984    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-openshift-ca\") pod \"93eae285-b8d0-4f49-b72a-d6de913834ee\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") "
Oct 30 23:41:45.924169 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:45.924157    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-xks-ca\") pod \"93eae285-b8d0-4f49-b72a-d6de913834ee\" (UID: \"93eae285-b8d0-4f49-b72a-d6de913834ee\") "
Oct 30 23:41:45.924506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:45.924452    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-openshift-ca" pod "93eae285-b8d0-4f49-b72a-d6de913834ee" (UID: "93eae285-b8d0-4f49-b72a-d6de913834ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:41:45.925093 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:45.925064    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:45.926698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:45.926670    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93eae285-b8d0-4f49-b72a-d6de913834ee-kube-api-access-v6gsp" pod "93eae285-b8d0-4f49-b72a-d6de913834ee" (UID: "93eae285-b8d0-4f49-b72a-d6de913834ee"). InnerVolumeSpecName "kube-api-access-v6gsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:41:45.926886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:45.926865    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-xks-ca" pod "93eae285-b8d0-4f49-b72a-d6de913834ee" (UID: "93eae285-b8d0-4f49-b72a-d6de913834ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:41:46.026645 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:46.026617    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93eae285-b8d0-4f49-b72a-d6de913834ee-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:46.026804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:46.026664    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6gsp\" (UniqueName: \"kubernetes.io/projected/93eae285-b8d0-4f49-b72a-d6de913834ee-kube-api-access-v6gsp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:46.700076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:46.700039    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"
Oct 30 23:41:46.700282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:46.700068    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg" event={"ID":"93eae285-b8d0-4f49-b72a-d6de913834ee","Type":"ContainerDied","Data":"23c81de3526ae2e4ddffe374a0265c413de6d80c15019c4cc1dbfdf178242bd3"}
Oct 30 23:41:46.700282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:46.700130    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c81de3526ae2e4ddffe374a0265c413de6d80c15019c4cc1dbfdf178242bd3"
Oct 30 23:41:54.837768 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.834288    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"]
Oct 30 23:41:54.837768 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.835282    4555 cpu_manager.go:414] "containerMap: removing container" podUID="93eae285-b8d0-4f49-b72a-d6de913834ee" containerName="revoke-keys"
Oct 30 23:41:54.837768 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.835306    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93eae285-b8d0-4f49-b72a-d6de913834ee" containerName="revoke-keys"
Oct 30 23:41:54.837768 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.835491    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="93eae285-b8d0-4f49-b72a-d6de913834ee" containerName="revoke-keys"
Oct 30 23:41:54.845610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.845582    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"]
Oct 30 23:41:54.845941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.845916    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:54.850576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.850559    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:41:54.851622 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.851599    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:41:54.865718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.863558    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59ft\" (UniqueName: \"kubernetes.io/projected/9dfa9cfa-d1f1-4589-80c7-71409236eef8-kube-api-access-x59ft\") pod \"maas-api-revoke-sub-ba7a0243-z44vz\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:54.865718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.863704    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba7a0243-z44vz\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:54.865718 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.863879    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba7a0243-z44vz\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:54.965794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.965756    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba7a0243-z44vz\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:54.966008 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.965940    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59ft\" (UniqueName: \"kubernetes.io/projected/9dfa9cfa-d1f1-4589-80c7-71409236eef8-kube-api-access-x59ft\") pod \"maas-api-revoke-sub-ba7a0243-z44vz\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:54.966074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.966025    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba7a0243-z44vz\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:54.966613 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.966579    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba7a0243-z44vz\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:54.969127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.969100    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba7a0243-z44vz\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:54.978748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:54.978725    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59ft\" (UniqueName: \"kubernetes.io/projected/9dfa9cfa-d1f1-4589-80c7-71409236eef8-kube-api-access-x59ft\") pod \"maas-api-revoke-sub-ba7a0243-z44vz\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:55.163706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.163672    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:55.455796 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.455773    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"]
Oct 30 23:41:55.457184 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:41:55.457156    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfa9cfa_d1f1_4589_80c7_71409236eef8.slice/crio-82c328e87cd4a86bf07465cbd0b6de30c4b6a06d61264b0ac4f086d413233515 WatchSource:0}: Error finding container 82c328e87cd4a86bf07465cbd0b6de30c4b6a06d61264b0ac4f086d413233515: Status 404 returned error can't find the container with id 82c328e87cd4a86bf07465cbd0b6de30c4b6a06d61264b0ac4f086d413233515
Oct 30 23:41:55.770088 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.770050    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz" event={"ID":"9dfa9cfa-d1f1-4589-80c7-71409236eef8","Type":"ContainerStarted","Data":"82c328e87cd4a86bf07465cbd0b6de30c4b6a06d61264b0ac4f086d413233515"}
Oct 30 23:41:55.860836 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.860799    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"]
Oct 30 23:41:55.867801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.867772    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:55.916353 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.916326    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"]
Oct 30 23:41:55.994062 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.994034    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:55.994207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.994082    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:55.994207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.994152    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:55.994292 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.994239    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrvhf\" (UniqueName: \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-kube-api-access-jrvhf\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:55.994292 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.994265    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:55.994397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.994350    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5625facc-0053-4711-ae40-3e0f5d9fabf8-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:55.994450 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.994426    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:55.994490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.994463    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:55.994552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:55.994541    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096159    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096223    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096259    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096281    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096307    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrvhf\" (UniqueName: \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-kube-api-access-jrvhf\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096323    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096434    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5625facc-0053-4711-ae40-3e0f5d9fabf8-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096470    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096520    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096956 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096653    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.096956 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.096920    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.097202 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.097181    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.097282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.097243    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.097341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.097313    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5625facc-0053-4711-ae40-3e0f5d9fabf8-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.100509 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.100479    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.100747 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.100727    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.114876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.114857    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.115131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.115114    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrvhf\" (UniqueName: \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-kube-api-access-jrvhf\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.206970 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.206935    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:56.402116 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.402023    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"]
Oct 30 23:41:56.408268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.408238    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.416257 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.416230    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"]
Oct 30 23:41:56.503688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.503655    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.503878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.503704    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.503878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.503829    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.504017 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.504002    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.504056 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.504043    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.504111 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.504074    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.504202 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.504177    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xck\" (UniqueName: \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-kube-api-access-z7xck\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.504276 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.504262    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.504327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.504291    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.583022 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.581489    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"]
Oct 30 23:41:56.594132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.594094    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:56.594495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.594473    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:56.594680 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.594653    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:56.614912 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.614880    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.615162 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.615129    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.615346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.615331    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.615624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.615605    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.615781 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.615765    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.615892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.615877    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.616016 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.615982    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xck\" (UniqueName: \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-kube-api-access-z7xck\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.616156 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.616143    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.616247 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.616234    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.644396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.626651    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.644667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.627872    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.644777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.628110    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.644852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.631705    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.644932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.632308    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.645036 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.634667    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.645130 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.640037    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.645223 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.643219    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xck\" (UniqueName: \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-kube-api-access-z7xck\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.650084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.650034    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.727900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.727650    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:56.798843 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.798804    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" event={"ID":"5625facc-0053-4711-ae40-3e0f5d9fabf8","Type":"ContainerStarted","Data":"1bbbaa93df5f304d4353779b02aeba208f816147f6038b069025749ba2d81ca5"}
Oct 30 23:41:56.799056 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.798875    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" event={"ID":"5625facc-0053-4711-ae40-3e0f5d9fabf8","Type":"ContainerStarted","Data":"039a8fb3a6c380829840ddefd15ecfbe4d934af51cb484142fb52e062709e904"}
Oct 30 23:41:56.802578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.802546    4555 generic.go:356] "Generic (PLEG): container finished" podID="9dfa9cfa-d1f1-4589-80c7-71409236eef8" containerID="7c2eb48cc6f234c3c0af3bfdd7ebe9b78f55241ce9eff95ee78318450684997a" exitCode=0
Oct 30 23:41:56.803039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.803009    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz" event={"ID":"9dfa9cfa-d1f1-4589-80c7-71409236eef8","Type":"ContainerDied","Data":"7c2eb48cc6f234c3c0af3bfdd7ebe9b78f55241ce9eff95ee78318450684997a"}
Oct 30 23:41:56.899410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:56.892921    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podStartSLOduration=1.8928984949999998 podStartE2EDuration="1.892898495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:56.829951314 +0000 UTC m=+1375.756588897" watchObservedRunningTime="2026-10-30 23:41:56.892898495 +0000 UTC m=+1375.819536079"
Oct 30 23:41:57.596524 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:57.596489    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:41:57.596804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:57.596771    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:41:57.664188 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:57.664146    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:41:57.764230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:57.764188    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"]
Oct 30 23:41:57.953908 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:57.952350    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:57.953908 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:57.952492    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:57.953908 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:57.952539    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:41:58.148447 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:41:58.144624    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:41:58.211249 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:58.210793    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:41:58.211249 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:58.210878    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:41:58.911600 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:58.911557    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:58.912266 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:58.912238    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:41:58.999105 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:58.999010    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:41:59.006947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.006917    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:41:59.013509 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.011640    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" event={"ID":"4d8379c8-e077-4d6a-a325-a91ce3c911bb","Type":"ContainerStarted","Data":"8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f"}
Oct 30 23:41:59.021396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.017865    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" event={"ID":"4d8379c8-e077-4d6a-a325-a91ce3c911bb","Type":"ContainerStarted","Data":"cddf4fc2af4503281c6acaddfe963f6f415fe6a2856970a0a36a7b870edd0424"}
Oct 30 23:41:59.025528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.025452    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz" event={"ID":"9dfa9cfa-d1f1-4589-80c7-71409236eef8","Type":"ContainerDied","Data":"82c328e87cd4a86bf07465cbd0b6de30c4b6a06d61264b0ac4f086d413233515"}
Oct 30 23:41:59.035501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.035457    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c328e87cd4a86bf07465cbd0b6de30c4b6a06d61264b0ac4f086d413233515"
Oct 30 23:41:59.090410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.071881    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podStartSLOduration=3.071861002 podStartE2EDuration="3.071861002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:59.062345099 +0000 UTC m=+1377.988982704" watchObservedRunningTime="2026-10-30 23:41:59.071861002 +0000 UTC m=+1377.998498583"
Oct 30 23:41:59.103443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.103412    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-xks-ca\") pod \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") "
Oct 30 23:41:59.107619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.103753    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-openshift-ca\") pod \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") "
Oct 30 23:41:59.107931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.107911    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dfa9cfa-d1f1-4589-80c7-71409236eef8-kube-api-access-x59ft\" (UniqueName: \"kubernetes.io/projected/9dfa9cfa-d1f1-4589-80c7-71409236eef8-kube-api-access-x59ft\") pod \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\" (UID: \"9dfa9cfa-d1f1-4589-80c7-71409236eef8\") "
Oct 30 23:41:59.113513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.107581    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-openshift-ca" pod "9dfa9cfa-d1f1-4589-80c7-71409236eef8" (UID: "9dfa9cfa-d1f1-4589-80c7-71409236eef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:41:59.141408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.130142    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfa9cfa-d1f1-4589-80c7-71409236eef8-kube-api-access-x59ft" pod "9dfa9cfa-d1f1-4589-80c7-71409236eef8" (UID: "9dfa9cfa-d1f1-4589-80c7-71409236eef8"). InnerVolumeSpecName "kube-api-access-x59ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:41:59.141774 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.141716    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-xks-ca" pod "9dfa9cfa-d1f1-4589-80c7-71409236eef8" (UID: "9dfa9cfa-d1f1-4589-80c7-71409236eef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:41:59.371576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.349602    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:41:59.371576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.349683    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:41:59.514528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.484711    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:59.514528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.495567    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:59.514528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.495607    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dfa9cfa-d1f1-4589-80c7-71409236eef8-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:59.514528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.495624    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x59ft\" (UniqueName: \"kubernetes.io/projected/9dfa9cfa-d1f1-4589-80c7-71409236eef8-kube-api-access-x59ft\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:41:59.552411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.539576    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:59.851409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.841514    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:41:59.851409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.843986    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:41:59.851409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.851303    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:59.951397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.920670    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:41:59.951397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.929590    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:41:59.951397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.929637    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:41:59.951397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.950161    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:41:59.951397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:41:59.950407    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:42:00.020397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.007768    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:42:00.042559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.042507    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:00.057555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.057504    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:00.058273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.058253    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:42:00.106865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.106768    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Oct 30 23:42:00.208397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.176314    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"
Oct 30 23:42:00.285517 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.285457    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:00.285813 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.285766    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:00.523454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.497852    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:00.548396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.545875    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:00.548396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.546508    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:42:00.548396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.546775    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:42:00.858453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.741860    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:00.858453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.741921    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:00.917405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.882472    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:00.917405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.883293    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:42:00.925535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:00.925492    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:01.020162 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.020120    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:01.030397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.021166    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:42:01.074521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.073462    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:42:01.075782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.073513    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:01.077184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.075397    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:01.077654 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.077633    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:42:01.311295 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.311257    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:01.313549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.313517    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:01.547884 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.547845    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:01.549168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.549137    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 23:42:01.724867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.724769    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:01.769705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.768730    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:01.769705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.768801    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:01.882810 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.882770    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:42:01.932553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.932509    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:01.964134 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:01.964095    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:42:02.026850 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.026815    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:42:02.159424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.144513    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:42:02.301773 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.160165    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:02.303324 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.160236    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:02.311816 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.227113    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:02.312073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.312039    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:02.557588 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.557482    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:02.558898 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.558862    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:42:02.729780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.729740    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:02.730050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.730021    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:02.842544 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.842444    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:02.887399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.884506    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:42:02.966517 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:02.966467    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:03.028500 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.028459    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:03.226952 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.226858    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:03.227738 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.227702    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:03.337396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.335682    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:03.337396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.336029    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:42:03.358405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.356879    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:42:03.583395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.558608    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 23:42:03.644395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.563465    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:03.644395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.641462    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:03.745734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.745187    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:03.745734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.745250    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:03.915451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.915140    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:42:03.969687 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:03.968487    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:04.043427 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.036768    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:42:04.208821 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.208728    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:04.209889 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.209856    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:04.337592 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.337544    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:04.339501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.339466    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:04.361530 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.361485    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 30 23:42:04.361847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.361820    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:42:04.660787 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.660744    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:04.691517 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.661089    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:42:04.726409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.725483    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:04.726409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.725549    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:04.752084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.748336    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:04.752084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.748419    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:04.916576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.916467    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:04.970930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:04.970886    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:05.048396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.047515    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:42:05.128881 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.128499    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:05.128881 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.128568    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:05.251008 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.247789    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:05.251008 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.247849    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:05.252603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.252575    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:05.252863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.252808    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:05.318722 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.318686    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:05.319012 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.318977    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:05.346041 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.346000    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:05.348810 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.348770    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:05.375601 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.375558    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:42:05.554342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.554289    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:05.554726 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.554693    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:05.687187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.684788    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:05.688018 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.684719    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:05.729065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.729033    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:05.729409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.729344    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:05.767235 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.767204    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:42:05.932329 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.932233    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:42:05.948404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.947982    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:05.964137 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.962168    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:42:06.016427 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:05.982428    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:06.292317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.292266    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:42:06.395449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.378320    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:42:06.395449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.379800    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:06.395449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.379856    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:06.395449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.382744    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:06.410419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.400597    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:42:06.528436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.492276    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:42:06.679489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.672532    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:42:06.807731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.807687    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:06.807991 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.807960    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:06.990452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:06.989496    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:07.147542 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:07.143746    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:42:07.152161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:07.150032    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:42:07.232405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:07.232336    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:07.233084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:07.232686    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:07.728877 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:07.728837    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:07.729400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:07.728914    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:08.217455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:08.213979    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:08.217455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:08.214040    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:08.728941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:08.728900    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:08.729416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:08.728970    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:09.225388 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:09.225326    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:09.225608 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:09.225432    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:09.729350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:09.729298    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:09.729802 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:09.729362    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:10.207748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:10.207715    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:10.208005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:10.207983    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:10.729231 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:10.729197    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:10.729455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:10.729252    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:10.918234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:10.918187    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"]
Oct 30 23:42:10.920741 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:10.920710    4555 cpu_manager.go:414] "containerMap: removing container" podUID="9dfa9cfa-d1f1-4589-80c7-71409236eef8" containerName="revoke-keys"
Oct 30 23:42:10.920998 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:10.920974    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dfa9cfa-d1f1-4589-80c7-71409236eef8" containerName="revoke-keys"
Oct 30 23:42:10.921311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:10.921296    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dfa9cfa-d1f1-4589-80c7-71409236eef8" containerName="revoke-keys"
Oct 30 23:42:10.928508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:10.928486    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:10.960232 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:10.960188    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"]
Oct 30 23:42:11.047738 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.047708    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.047983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.047958    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.048070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.048003    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.048170 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.048142    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.048229 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.048209    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.048272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.048248    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp585\" (UniqueName: \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-kube-api-access-rp585\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.048424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.048360    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.048498 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.048481    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.048562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.048547    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.149661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.149626    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.149880 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.149767    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.149880 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.149819    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.149880 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.149861    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.150080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.149914    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.150080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.149941    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.150080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.149978    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp585\" (UniqueName: \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-kube-api-access-rp585\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.150080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.150024    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.150290 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.150087    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.150446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.150424    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-credential-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.150446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.150435    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-socket\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.150597 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.150481    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-data\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.150849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.150822    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istiod-ca-cert\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.151282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.151252    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-certs\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.153519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.153489    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-podinfo\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.154109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.154074    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-envoy\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.183397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.179782    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp585\" (UniqueName: \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-kube-api-access-rp585\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.203397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.195879    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-token\") pod \"e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") " pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.208169 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.207792    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 23:42:11.208169 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.207850    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 23:42:11.217021 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.216987    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:42:11.317614 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.317479    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:11.677849 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:42:11.677810    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa094ff_af62_4f11_a3b9_d2f08e4d003c.slice/crio-128864b7a5f6ad92820081a0f665207771c48f425a785130619f8c7ddd1971d1 WatchSource:0}: Error finding container 128864b7a5f6ad92820081a0f665207771c48f425a785130619f8c7ddd1971d1: Status 404 returned error can't find the container with id 128864b7a5f6ad92820081a0f665207771c48f425a785130619f8c7ddd1971d1
Oct 30 23:42:11.680819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.680790    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"]
Oct 30 23:42:11.684753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.684711    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:42:11.684859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.684805    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:42:11.684859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.684851    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:42:11.728410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.728358    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:42:11.729243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.729041    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:42:11.729243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:11.729092    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:42:12.172847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:12.172810    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" event={"ID":"caa094ff-af62-4f11-a3b9-d2f08e4d003c","Type":"ContainerStarted","Data":"f5c54d3e9b8f61f80903441ec1b3cc7ec52341ff66ae88bd17c2c2d31749aad1"}
Oct 30 23:42:12.173329 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:12.172871    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" event={"ID":"caa094ff-af62-4f11-a3b9-d2f08e4d003c","Type":"ContainerStarted","Data":"128864b7a5f6ad92820081a0f665207771c48f425a785130619f8c7ddd1971d1"}
Oct 30 23:42:12.212824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:12.212796    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:42:12.213961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:12.213942    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:42:12.222080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:12.222040    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podStartSLOduration=2.222025166 podStartE2EDuration="2.222025166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:12.219580972 +0000 UTC m=+1391.146218553" watchObservedRunningTime="2026-10-30 23:42:12.222025166 +0000 UTC m=+1391.148662748"
Oct 30 23:42:12.318121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:12.318089    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:12.320220 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:12.320191    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 23:42:12.320406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:12.320250    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 23:42:13.123894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:13.123492    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:42:13.132725 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:13.132400    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:42:13.318991 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:13.318956    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 23:42:13.319411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:13.319020    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 23:42:14.318571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:14.318538    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 23:42:14.318746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:14.318590    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 23:42:15.318518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:15.318489    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 23:42:15.318884 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:15.318540    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 23:42:15.914957 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:15.914920    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:42:16.329974 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:16.329938    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 23:42:16.331701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:16.331658    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 23:42:16.823398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:16.822788    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"]
Oct 30 23:42:16.867502 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:16.867466    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"]
Oct 30 23:42:16.867921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:16.867835    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:16.880761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:16.880731    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:42:16.956539 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:16.956499    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxsxt\" (UniqueName: \"kubernetes.io/projected/e1859d0e-496f-49d0-ad41-aa1a84cc424f-kube-api-access-mxsxt\") pod \"maas-api-revoke-sub-0b62f526-n6hhs\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:16.956836 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:16.956818    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b62f526-n6hhs\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:16.957092 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:16.957074    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b62f526-n6hhs\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:16.982694 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:16.982661    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:42:17.065780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:17.058906    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxsxt\" (UniqueName: \"kubernetes.io/projected/e1859d0e-496f-49d0-ad41-aa1a84cc424f-kube-api-access-mxsxt\") pod \"maas-api-revoke-sub-0b62f526-n6hhs\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:17.066519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:17.066494    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b62f526-n6hhs\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:17.067702 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:17.067675    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b62f526-n6hhs\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:17.081547 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:17.067444    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b62f526-n6hhs\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:17.103659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:17.103626    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b62f526-n6hhs\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:17.106411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:17.106385    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxsxt\" (UniqueName: \"kubernetes.io/projected/e1859d0e-496f-49d0-ad41-aa1a84cc424f-kube-api-access-mxsxt\") pod \"maas-api-revoke-sub-0b62f526-n6hhs\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:17.231593 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:17.231562    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:17.352769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:17.345403    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 23:42:17.352769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:17.345463    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 23:42:17.980561 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:17.940285    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"]
Oct 30 23:42:18.101601 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:42:18.071463    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1859d0e_496f_49d0_ad41_aa1a84cc424f.slice/crio-87aef1d8e329bf7b3d875f42720a325a3304dd0ca4723dfde1f3c5fe89550564 WatchSource:0}: Error finding container 87aef1d8e329bf7b3d875f42720a325a3304dd0ca4723dfde1f3c5fe89550564: Status 404 returned error can't find the container with id 87aef1d8e329bf7b3d875f42720a325a3304dd0ca4723dfde1f3c5fe89550564
Oct 30 23:42:18.318922 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:18.318888    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 23:42:18.319219 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:18.319173    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 23:42:18.427402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:18.420712    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs" event={"ID":"e1859d0e-496f-49d0-ad41-aa1a84cc424f","Type":"ContainerStarted","Data":"87aef1d8e329bf7b3d875f42720a325a3304dd0ca4723dfde1f3c5fe89550564"}
Oct 30 23:42:19.001886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:19.000470    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:42:19.078396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:19.075276    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:19.083837 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:19.083798    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:19.304695 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:19.304648    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:42:19.321396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:19.319886    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 23:42:19.321396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:19.319947    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 23:42:19.342500 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:19.342454    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:19.490885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:19.490847    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs" event={"ID":"e1859d0e-496f-49d0-ad41-aa1a84cc424f","Type":"ContainerStarted","Data":"85d131cdfb2af6aac94abb97dd6edda4d049a10216ea0b5e2e8b1dd69e53b6f5"}
Oct 30 23:42:19.554395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:19.553453    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:19.587895 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:19.587832    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs" podStartSLOduration=3.587810184 podStartE2EDuration="3.587810184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:19.567116605 +0000 UTC m=+1398.493754187" watchObservedRunningTime="2026-10-30 23:42:19.587810184 +0000 UTC m=+1398.514447766"
Oct 30 23:42:20.182575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.182534    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:20.182842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.182817    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:20.186132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.186100    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:20.343412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.340094    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 23:42:20.343412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.340167    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 23:42:20.395416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.391247    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 23:42:20.395416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.391788    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:42:20.406564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.406530    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:20.406791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.406760    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:20.482416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.482283    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:42:20.523766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.523733    4555 generic.go:356] "Generic (PLEG): container finished" podID="e1859d0e-496f-49d0-ad41-aa1a84cc424f" containerID="85d131cdfb2af6aac94abb97dd6edda4d049a10216ea0b5e2e8b1dd69e53b6f5" exitCode=0
Oct 30 23:42:20.525493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.525465    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs" event={"ID":"e1859d0e-496f-49d0-ad41-aa1a84cc424f","Type":"ContainerDied","Data":"85d131cdfb2af6aac94abb97dd6edda4d049a10216ea0b5e2e8b1dd69e53b6f5"}
Oct 30 23:42:20.904622 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.903950    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:42:20.953664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:20.953608    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:21.187454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:21.187050    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:21.342423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:21.318937    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 23:42:21.342423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:21.318994    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 23:42:21.384415 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:21.384154    4555 patch_prober.go:29] interesting pod/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:21.384415 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:21.384220    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:21.633962 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:21.633925    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:21.654731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:21.654678    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:21.692938 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:21.692877    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:22.189945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.189900    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:22.201539 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.201502    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:42:22.684396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.678792    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:42:22.684396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.680650    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:42:22.833540 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.821704    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 23:42:22.835630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.835603    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:42:22.841393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.839444    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:22.841698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.841667    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:22.873103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.873070    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:42:22.911401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.892825    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:22.963449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:22.952801    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:42:23.079388 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.078519    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:23.129720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.129687    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:42:23.138782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.138753    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:42:23.252493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.247444    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-xks-ca\") pod \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") "
Oct 30 23:42:23.252493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.247521    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1859d0e-496f-49d0-ad41-aa1a84cc424f-kube-api-access-mxsxt\" (UniqueName: \"kubernetes.io/projected/e1859d0e-496f-49d0-ad41-aa1a84cc424f-kube-api-access-mxsxt\") pod \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") "
Oct 30 23:42:23.252493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.247659    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-openshift-ca\") pod \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\" (UID: \"e1859d0e-496f-49d0-ad41-aa1a84cc424f\") "
Oct 30 23:42:23.255099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.255064    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-openshift-ca" pod "e1859d0e-496f-49d0-ad41-aa1a84cc424f" (UID: "e1859d0e-496f-49d0-ad41-aa1a84cc424f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:23.255963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.255932    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.306874 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.285530    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1859d0e-496f-49d0-ad41-aa1a84cc424f-kube-api-access-mxsxt" pod "e1859d0e-496f-49d0-ad41-aa1a84cc424f" (UID: "e1859d0e-496f-49d0-ad41-aa1a84cc424f"). InnerVolumeSpecName "kube-api-access-mxsxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:23.307497 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.307462    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-xks-ca" pod "e1859d0e-496f-49d0-ad41-aa1a84cc424f" (UID: "e1859d0e-496f-49d0-ad41-aa1a84cc424f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:23.337973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.337797    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:23.337973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.337900    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:23.374151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.374116    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1859d0e-496f-49d0-ad41-aa1a84cc424f-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.374342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.374260    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxsxt\" (UniqueName: \"kubernetes.io/projected/e1859d0e-496f-49d0-ad41-aa1a84cc424f-kube-api-access-mxsxt\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.917521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.917479    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs" event={"ID":"e1859d0e-496f-49d0-ad41-aa1a84cc424f","Type":"ContainerDied","Data":"87aef1d8e329bf7b3d875f42720a325a3304dd0ca4723dfde1f3c5fe89550564"}
Oct 30 23:42:23.918073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.917540    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87aef1d8e329bf7b3d875f42720a325a3304dd0ca4723dfde1f3c5fe89550564"
Oct 30 23:42:23.918339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:23.918315    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"
Oct 30 23:42:24.378556 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:24.378504    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:24.378792 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:24.378561    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:24.714797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:24.714708    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:24.715359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:24.715337    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:24.717473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:24.717448    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 30 23:42:25.128948 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.128917    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"]
Oct 30 23:42:25.129666 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.129644    4555 cpu_manager.go:414] "containerMap: removing container" podUID="e1859d0e-496f-49d0-ad41-aa1a84cc424f" containerName="revoke-keys"
Oct 30 23:42:25.129666 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.129664    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1859d0e-496f-49d0-ad41-aa1a84cc424f" containerName="revoke-keys"
Oct 30 23:42:25.129826 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.129774    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1859d0e-496f-49d0-ad41-aa1a84cc424f" containerName="revoke-keys"
Oct 30 23:42:25.133632 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.133614    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.141090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.141068    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:42:25.141090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.141069    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:42:25.145945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.145922    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"]
Oct 30 23:42:25.218515 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.218487    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt8xn\" (UniqueName: \"kubernetes.io/projected/72b501a4-4b1f-4e82-8000-9ea14bb48386-kube-api-access-pt8xn\") pod \"maas-api-revoke-sub-c44fb6a5-xt494\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.218742 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.218723    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44fb6a5-xt494\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.218822 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.218756    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44fb6a5-xt494\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.320020 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.319987    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt8xn\" (UniqueName: \"kubernetes.io/projected/72b501a4-4b1f-4e82-8000-9ea14bb48386-kube-api-access-pt8xn\") pod \"maas-api-revoke-sub-c44fb6a5-xt494\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.320221 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.320202    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44fb6a5-xt494\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.320357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.320334    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44fb6a5-xt494\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.320865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.320840    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44fb6a5-xt494\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.322816 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.322798    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44fb6a5-xt494\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.341306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.341279    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt8xn\" (UniqueName: \"kubernetes.io/projected/72b501a4-4b1f-4e82-8000-9ea14bb48386-kube-api-access-pt8xn\") pod \"maas-api-revoke-sub-c44fb6a5-xt494\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.450117 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.450033    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:25.753506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.753476    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"]
Oct 30 23:42:25.754342 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:42:25.754304    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b501a4_4b1f_4e82_8000_9ea14bb48386.slice/crio-70977f1c38c54c276ad7f350800113a240b235c0dcfde5bd1ebf8206db6484a7 WatchSource:0}: Error finding container 70977f1c38c54c276ad7f350800113a240b235c0dcfde5bd1ebf8206db6484a7: Status 404 returned error can't find the container with id 70977f1c38c54c276ad7f350800113a240b235c0dcfde5bd1ebf8206db6484a7
Oct 30 23:42:25.873228 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.873190    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"]
Oct 30 23:42:25.884134 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.884087    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:25.905919 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.905865    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"]
Oct 30 23:42:25.931404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.929949    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:25.931404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.929997    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:25.931404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.930151    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:25.931404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.930216    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vwt\" (UniqueName: \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-kube-api-access-l6vwt\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:25.931404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.930358    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:25.931404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.930444    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f7cf65-6501-4af8-81af-9e2c0c123b57-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:25.931404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.930481    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:25.931404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.930514    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:25.931404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.930549    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:25.937876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:25.937850    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494" event={"ID":"72b501a4-4b1f-4e82-8000-9ea14bb48386","Type":"ContainerStarted","Data":"70977f1c38c54c276ad7f350800113a240b235c0dcfde5bd1ebf8206db6484a7"}
Oct 30 23:42:26.031834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.031802    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.031989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.031856    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vwt\" (UniqueName: \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-kube-api-access-l6vwt\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032087 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032040    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032171 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032114    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f7cf65-6501-4af8-81af-9e2c0c123b57-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032171 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032143    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032292 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032180    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032292 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032221    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032340    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032482 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032432    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032580 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032554    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-data\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032576    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032802 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032689    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-credential-socket\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032850 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032824    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-certs\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.032945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.032920    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f7cf65-6501-4af8-81af-9e2c0c123b57-istiod-ca-cert\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.034320 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.034300    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-envoy\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.034921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.034891    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-podinfo\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.049104 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.049073    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vwt\" (UniqueName: \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-kube-api-access-l6vwt\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.049750 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.049728    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-token\") pod \"e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") " pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.112253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.112225    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"]
Oct 30 23:42:26.116568 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.116553    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.160522 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.160490    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"]
Oct 30 23:42:26.213053 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.213019    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:26.235250 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.235217    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165550ee-3ccc-485a-bbe3-d7e7d295253d-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.235459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.235332    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.235459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.235419    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.235620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.235471    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7fwv\" (UniqueName: \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-kube-api-access-j7fwv\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.235620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.235505    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.235620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.235573    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.235869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.235740    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.235869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.235779    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.235993 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.235920    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.338295    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.338354    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.338398    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.338517    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.338618    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165550ee-3ccc-485a-bbe3-d7e7d295253d-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.338673    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.338706    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.338744    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7fwv\" (UniqueName: \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-kube-api-access-j7fwv\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.338767    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.338951    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.339993 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.339680    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-data\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.342056 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.341090    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-credential-socket\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.342056 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.341469    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-certs\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.342227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.342151    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-envoy\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.343353 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.343306    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165550ee-3ccc-485a-bbe3-d7e7d295253d-istiod-ca-cert\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.344761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.344734    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-podinfo\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.380291 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.380241    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-token\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.383606 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.383580    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7fwv\" (UniqueName: \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-kube-api-access-j7fwv\") pod \"e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") " pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.430310 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.430277    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:26.530457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.527919    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"]
Oct 30 23:42:26.531449 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:42:26.531400    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f7cf65_6501_4af8_81af_9e2c0c123b57.slice/crio-48a08d38b0c5fc236bb2c565341d15c56ba1a19d75934b3867ed830b163c7fe3 WatchSource:0}: Error finding container 48a08d38b0c5fc236bb2c565341d15c56ba1a19d75934b3867ed830b163c7fe3: Status 404 returned error can't find the container with id 48a08d38b0c5fc236bb2c565341d15c56ba1a19d75934b3867ed830b163c7fe3
Oct 30 23:42:26.534235 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.534174    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:42:26.534551 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.534512    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:42:26.534731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.534705    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:42:26.801619 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:42:26.801586    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165550ee_3ccc_485a_bbe3_d7e7d295253d.slice/crio-1419de60580e2a1012608e2cdbea13b41b6112a2d587b06172a4ea3e791d72c3 WatchSource:0}: Error finding container 1419de60580e2a1012608e2cdbea13b41b6112a2d587b06172a4ea3e791d72c3: Status 404 returned error can't find the container with id 1419de60580e2a1012608e2cdbea13b41b6112a2d587b06172a4ea3e791d72c3
Oct 30 23:42:26.802004 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.801985    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"]
Oct 30 23:42:26.804395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.804336    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:42:26.804480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.804444    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:42:26.804536 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.804491    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:42:26.951745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.951702    4555 generic.go:356] "Generic (PLEG): container finished" podID="72b501a4-4b1f-4e82-8000-9ea14bb48386" containerID="1266aa490793c0506a7bf8ecfe0f2e0d2cfafa1c74f931cb95657492145f2ee6" exitCode=0
Oct 30 23:42:26.951958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.951808    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494" event={"ID":"72b501a4-4b1f-4e82-8000-9ea14bb48386","Type":"ContainerDied","Data":"1266aa490793c0506a7bf8ecfe0f2e0d2cfafa1c74f931cb95657492145f2ee6"}
Oct 30 23:42:26.960579 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.960548    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" event={"ID":"44f7cf65-6501-4af8-81af-9e2c0c123b57","Type":"ContainerStarted","Data":"2967f7f09edd2710845679214633df20b74a56889fb9ec685882519e826102b7"}
Oct 30 23:42:26.960720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.960608    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" event={"ID":"44f7cf65-6501-4af8-81af-9e2c0c123b57","Type":"ContainerStarted","Data":"48a08d38b0c5fc236bb2c565341d15c56ba1a19d75934b3867ed830b163c7fe3"}
Oct 30 23:42:26.963248 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.963218    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" event={"ID":"165550ee-3ccc-485a-bbe3-d7e7d295253d","Type":"ContainerStarted","Data":"0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84"}
Oct 30 23:42:26.963344 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:26.963271    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" event={"ID":"165550ee-3ccc-485a-bbe3-d7e7d295253d","Type":"ContainerStarted","Data":"1419de60580e2a1012608e2cdbea13b41b6112a2d587b06172a4ea3e791d72c3"}
Oct 30 23:42:27.158483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:27.155513    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" podStartSLOduration=1.155492048 podStartE2EDuration="1.155492048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:27.093740464 +0000 UTC m=+1406.020378045" watchObservedRunningTime="2026-10-30 23:42:27.155492048 +0000 UTC m=+1406.082129645"
Oct 30 23:42:27.159096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:27.159057    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" podStartSLOduration=2.15904366 podStartE2EDuration="2.15904366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:27.155678384 +0000 UTC m=+1406.082315973" watchObservedRunningTime="2026-10-30 23:42:27.15904366 +0000 UTC m=+1406.085681240"
Oct 30 23:42:27.214140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:27.214104    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:27.215361 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:27.215328    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:42:27.215504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:27.215428    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 23:42:27.432330 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:27.432248    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:27.432836 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:27.432811    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 23:42:27.433046 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:27.433003    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 23:42:27.538621 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:42:27.538583    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:42:28.220177 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.220141    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:42:28.220780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.220747    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 23:42:28.446518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.444824    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 23:42:28.446518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.444881    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 23:42:28.561537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.561502    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:28.748725 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.748643    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-xks-ca\") pod \"72b501a4-4b1f-4e82-8000-9ea14bb48386\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") "
Oct 30 23:42:28.748865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.748699    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-openshift-ca\") pod \"72b501a4-4b1f-4e82-8000-9ea14bb48386\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") "
Oct 30 23:42:28.749046 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.749028    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b501a4-4b1f-4e82-8000-9ea14bb48386-kube-api-access-pt8xn\" (UniqueName: \"kubernetes.io/projected/72b501a4-4b1f-4e82-8000-9ea14bb48386-kube-api-access-pt8xn\") pod \"72b501a4-4b1f-4e82-8000-9ea14bb48386\" (UID: \"72b501a4-4b1f-4e82-8000-9ea14bb48386\") "
Oct 30 23:42:28.749254 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.749232    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-openshift-ca" pod "72b501a4-4b1f-4e82-8000-9ea14bb48386" (UID: "72b501a4-4b1f-4e82-8000-9ea14bb48386"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:28.749619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.749601    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:28.751452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.751422    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b501a4-4b1f-4e82-8000-9ea14bb48386-kube-api-access-pt8xn" pod "72b501a4-4b1f-4e82-8000-9ea14bb48386" (UID: "72b501a4-4b1f-4e82-8000-9ea14bb48386"). InnerVolumeSpecName "kube-api-access-pt8xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:28.751620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.751605    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-xks-ca" pod "72b501a4-4b1f-4e82-8000-9ea14bb48386" (UID: "72b501a4-4b1f-4e82-8000-9ea14bb48386"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:28.850461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.850423    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt8xn\" (UniqueName: \"kubernetes.io/projected/72b501a4-4b1f-4e82-8000-9ea14bb48386-kube-api-access-pt8xn\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:28.850461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:28.850447    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b501a4-4b1f-4e82-8000-9ea14bb48386-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:29.035246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.034690    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494" event={"ID":"72b501a4-4b1f-4e82-8000-9ea14bb48386","Type":"ContainerDied","Data":"70977f1c38c54c276ad7f350800113a240b235c0dcfde5bd1ebf8206db6484a7"}
Oct 30 23:42:29.035246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.034774    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70977f1c38c54c276ad7f350800113a240b235c0dcfde5bd1ebf8206db6484a7"
Oct 30 23:42:29.035246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.034862    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"
Oct 30 23:42:29.219261 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.216447    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:42:29.219261 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.216504    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 23:42:29.443270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.436025    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 23:42:29.443270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.436085    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 23:42:29.801411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.800948    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"]
Oct 30 23:42:29.819398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.808958    4555 cpu_manager.go:414] "containerMap: removing container" podUID="72b501a4-4b1f-4e82-8000-9ea14bb48386" containerName="revoke-keys"
Oct 30 23:42:29.819398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.809000    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b501a4-4b1f-4e82-8000-9ea14bb48386" containerName="revoke-keys"
Oct 30 23:42:29.819398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.809154    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b501a4-4b1f-4e82-8000-9ea14bb48386" containerName="revoke-keys"
Oct 30 23:42:29.849696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.848972    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:29.850450 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.850419    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"]
Oct 30 23:42:29.888345 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.884408    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:42:29.915397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.915048    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:42:29.925528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.925481    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a806a1a-z9ngh\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:29.925971 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.925947    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a806a1a-z9ngh\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:29.926108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:29.926092    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br24v\" (UniqueName: \"kubernetes.io/projected/d0a16667-9aa5-4686-9c72-98b099cb6282-kube-api-access-br24v\") pod \"maas-api-revoke-sub-2a806a1a-z9ngh\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:30.027603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.027565    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a806a1a-z9ngh\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:30.027801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.027699    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a806a1a-z9ngh\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:30.027801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.027727    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br24v\" (UniqueName: \"kubernetes.io/projected/d0a16667-9aa5-4686-9c72-98b099cb6282-kube-api-access-br24v\") pod \"maas-api-revoke-sub-2a806a1a-z9ngh\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:30.028975 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.028951    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a806a1a-z9ngh\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:30.030729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.030699    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a806a1a-z9ngh\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:30.042562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.042538    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br24v\" (UniqueName: \"kubernetes.io/projected/d0a16667-9aa5-4686-9c72-98b099cb6282-kube-api-access-br24v\") pod \"maas-api-revoke-sub-2a806a1a-z9ngh\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:30.214453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.214069    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:42:30.214453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.214159    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 23:42:30.234168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.234135    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:30.437864 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.437723    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 23:42:30.438107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.437884    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 23:42:30.628785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:30.628726    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"]
Oct 30 23:42:31.142414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:31.136609    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh" event={"ID":"d0a16667-9aa5-4686-9c72-98b099cb6282","Type":"ContainerStarted","Data":"874dbf40f4ecc0220656054819200f442d29434d0c4b8019fecf3b3c43818dbc"}
Oct 30 23:42:31.221579 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:31.221545    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:42:31.221688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:31.221615    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 23:42:31.430987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:31.430952    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 23:42:31.431196 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:31.431002    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 23:42:32.189181 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:32.189083    4555 generic.go:356] "Generic (PLEG): container finished" podID="d0a16667-9aa5-4686-9c72-98b099cb6282" containerID="01bd42e200c132c9fab68c36a651f0a20ad3a43dfd078314d67cd8a05d840ad0" exitCode=0
Oct 30 23:42:32.189835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:32.189813    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh" event={"ID":"d0a16667-9aa5-4686-9c72-98b099cb6282","Type":"ContainerDied","Data":"01bd42e200c132c9fab68c36a651f0a20ad3a43dfd078314d67cd8a05d840ad0"}
Oct 30 23:42:32.215803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:32.215768    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:42:32.216032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:32.216009    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 23:42:32.430920 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:32.430882    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 23:42:32.431121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:32.430951    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 23:42:33.226390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.216012    4555 patch_prober.go:29] interesting pod/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:42:33.226390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.216072    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 23:42:33.431630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.431455    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 23:42:33.431630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.431531    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 23:42:33.488389 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.488313    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:33.547429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.547400    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a16667-9aa5-4686-9c72-98b099cb6282-kube-api-access-br24v\" (UniqueName: \"kubernetes.io/projected/d0a16667-9aa5-4686-9c72-98b099cb6282-kube-api-access-br24v\") pod \"d0a16667-9aa5-4686-9c72-98b099cb6282\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") "
Oct 30 23:42:33.547607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.547465    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-xks-ca\") pod \"d0a16667-9aa5-4686-9c72-98b099cb6282\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") "
Oct 30 23:42:33.547607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.547505    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-openshift-ca\") pod \"d0a16667-9aa5-4686-9c72-98b099cb6282\" (UID: \"d0a16667-9aa5-4686-9c72-98b099cb6282\") "
Oct 30 23:42:33.547926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.547902    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-openshift-ca" pod "d0a16667-9aa5-4686-9c72-98b099cb6282" (UID: "d0a16667-9aa5-4686-9c72-98b099cb6282"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:33.548318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.548267    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:33.550753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.550723    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-xks-ca" pod "d0a16667-9aa5-4686-9c72-98b099cb6282" (UID: "d0a16667-9aa5-4686-9c72-98b099cb6282"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:33.550858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.550792    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a16667-9aa5-4686-9c72-98b099cb6282-kube-api-access-br24v" pod "d0a16667-9aa5-4686-9c72-98b099cb6282" (UID: "d0a16667-9aa5-4686-9c72-98b099cb6282"). InnerVolumeSpecName "kube-api-access-br24v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:33.649441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.649410    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br24v\" (UniqueName: \"kubernetes.io/projected/d0a16667-9aa5-4686-9c72-98b099cb6282-kube-api-access-br24v\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:33.649441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:33.649437    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a16667-9aa5-4686-9c72-98b099cb6282-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:34.262064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:34.262023    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh" event={"ID":"d0a16667-9aa5-4686-9c72-98b099cb6282","Type":"ContainerDied","Data":"874dbf40f4ecc0220656054819200f442d29434d0c4b8019fecf3b3c43818dbc"}
Oct 30 23:42:34.262611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:34.262089    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="874dbf40f4ecc0220656054819200f442d29434d0c4b8019fecf3b3c43818dbc"
Oct 30 23:42:34.262611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:34.262154    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"
Oct 30 23:42:34.639397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:34.639130    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:34.640122 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:34.640098    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:34.643192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:34.643165    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:42:34.880616 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:34.877545    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:34.880616 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:34.879243    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:34.886312 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:34.886261    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:42:41.328435 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:41.324508    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:42:41.531408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:41.530501    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:41.692457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:41.691476    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:41.885240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:41.883457    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:42:41.898968 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:41.887985    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:42.029086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.005481    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 30 23:42:42.029086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.007082    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:42:42.113405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.086095    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:42.139420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.136924    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 23:42:42.310485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.283633    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:42.327910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.327869    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:42.329549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.329525    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:42:42.462416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.461388    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:42.462416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.461457    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:42.659455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.605470    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:42.659455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.606143    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:42:42.868410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.862393    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:42.885492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.885447    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:42.905864 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.905834    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:42:42.907194 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.907165    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:42.907736 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:42.907718    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:42:43.010610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.010516    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:42:43.018906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.012044    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:42:43.086748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.086697    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:43.113950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.113908    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:42:43.147969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.146736    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:43.148904 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.148532    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:42:43.302558 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.302513    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:43.356941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.354452    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:43.608986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.608885    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:43.643127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.643079    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 23:42:43.702866 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.702832    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:42:43.943445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.918849    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:42:43.944842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:43.944418    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:44.018552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.018510    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:42:44.142394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.141505    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:42:44.180808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.180765    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:44.330339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.317573    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:44.372555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.372492    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:44.394894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.394850    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:44.448316 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.438136    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"]
Oct 30 23:42:44.449799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.449764    4555 cpu_manager.go:414] "containerMap: removing container" podUID="d0a16667-9aa5-4686-9c72-98b099cb6282" containerName="revoke-keys"
Oct 30 23:42:44.449983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.449968    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a16667-9aa5-4686-9c72-98b099cb6282" containerName="revoke-keys"
Oct 30 23:42:44.450577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.450557    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a16667-9aa5-4686-9c72-98b099cb6282" containerName="revoke-keys"
Oct 30 23:42:44.618613 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.618510    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:44.627933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.627907    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:44.634971 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.634948    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:42:44.635334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.635318    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:42:44.653649 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.653602    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"]
Oct 30 23:42:44.720395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.705481    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4dm\" (UniqueName: \"kubernetes.io/projected/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-kube-api-access-tc4dm\") pod \"maas-api-revoke-sub-f5af5fce-w4vnj\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:44.720395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.705553    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5af5fce-w4vnj\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:44.720395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.705625    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5af5fce-w4vnj\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:44.831405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.816509    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4dm\" (UniqueName: \"kubernetes.io/projected/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-kube-api-access-tc4dm\") pod \"maas-api-revoke-sub-f5af5fce-w4vnj\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:44.831405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.816584    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5af5fce-w4vnj\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:44.831405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.816654    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5af5fce-w4vnj\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:44.831405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.817608    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5af5fce-w4vnj\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:44.895641 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.877741    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5af5fce-w4vnj\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:44.975455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.934937    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:42:44.999959 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:44.978765    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:45.033717 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.023851    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:42:45.073990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.072510    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:45.150051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.107090    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4dm\" (UniqueName: \"kubernetes.io/projected/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-kube-api-access-tc4dm\") pod \"maas-api-revoke-sub-f5af5fce-w4vnj\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:45.150051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.110244    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:42:45.151075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.150403    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:45.262691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.260072    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:42:45.262691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.260137    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:42:45.262691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.260206    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:45.262691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.261999    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:42:45.281936 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.281911    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:42:45.336397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.329494    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:45.411248 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.392004    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:42:45.792307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:45.792268    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"]
Oct 30 23:42:45.794497 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:42:45.794471    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba3b5bf_02e9_43a9_830e_6aecc3162d0c.slice/crio-0779b720542efa369eebac415a07f5192a47a31647345a077ae0cc735bcb8bb3 WatchSource:0}: Error finding container 0779b720542efa369eebac415a07f5192a47a31647345a077ae0cc735bcb8bb3: Status 404 returned error can't find the container with id 0779b720542efa369eebac415a07f5192a47a31647345a077ae0cc735bcb8bb3
Oct 30 23:42:46.294406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:46.294340    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj" event={"ID":"eba3b5bf-02e9-43a9-830e-6aecc3162d0c","Type":"ContainerStarted","Data":"eb08ffe1eede148e1713bcc713c6c17bffa4d00d7739701c1e1f6af048d74521"}
Oct 30 23:42:46.294585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:46.294419    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj" event={"ID":"eba3b5bf-02e9-43a9-830e-6aecc3162d0c","Type":"ContainerStarted","Data":"0779b720542efa369eebac415a07f5192a47a31647345a077ae0cc735bcb8bb3"}
Oct 30 23:42:46.325402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:46.319690    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj" podStartSLOduration=3.319670233 podStartE2EDuration="3.319670233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:46.317075297 +0000 UTC m=+1425.243712887" watchObservedRunningTime="2026-10-30 23:42:46.319670233 +0000 UTC m=+1425.246307813"
Oct 30 23:42:47.304901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:47.304863    4555 generic.go:356] "Generic (PLEG): container finished" podID="eba3b5bf-02e9-43a9-830e-6aecc3162d0c" containerID="eb08ffe1eede148e1713bcc713c6c17bffa4d00d7739701c1e1f6af048d74521" exitCode=0
Oct 30 23:42:47.305361 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:47.305053    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj" event={"ID":"eba3b5bf-02e9-43a9-830e-6aecc3162d0c","Type":"ContainerDied","Data":"eb08ffe1eede148e1713bcc713c6c17bffa4d00d7739701c1e1f6af048d74521"}
Oct 30 23:42:48.483698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:48.483670    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:48.538848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:48.538819    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-kube-api-access-tc4dm\" (UniqueName: \"kubernetes.io/projected/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-kube-api-access-tc4dm\") pod \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") "
Oct 30 23:42:48.539059 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:48.538917    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-xks-ca\") pod \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") "
Oct 30 23:42:48.539059 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:48.539022    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-openshift-ca\") pod \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\" (UID: \"eba3b5bf-02e9-43a9-830e-6aecc3162d0c\") "
Oct 30 23:42:48.539636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:48.539600    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-openshift-ca" pod "eba3b5bf-02e9-43a9-830e-6aecc3162d0c" (UID: "eba3b5bf-02e9-43a9-830e-6aecc3162d0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:48.541177 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:48.541144    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-kube-api-access-tc4dm" pod "eba3b5bf-02e9-43a9-830e-6aecc3162d0c" (UID: "eba3b5bf-02e9-43a9-830e-6aecc3162d0c"). InnerVolumeSpecName "kube-api-access-tc4dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:48.541807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:48.541777    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-xks-ca" pod "eba3b5bf-02e9-43a9-830e-6aecc3162d0c" (UID: "eba3b5bf-02e9-43a9-830e-6aecc3162d0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:48.641513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:48.641420    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:48.641513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:48.641456    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:48.641513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:48.641472    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc4dm\" (UniqueName: \"kubernetes.io/projected/eba3b5bf-02e9-43a9-830e-6aecc3162d0c-kube-api-access-tc4dm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:49.323587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:49.323556    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"
Oct 30 23:42:49.323779 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:49.323555    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj" event={"ID":"eba3b5bf-02e9-43a9-830e-6aecc3162d0c","Type":"ContainerDied","Data":"0779b720542efa369eebac415a07f5192a47a31647345a077ae0cc735bcb8bb3"}
Oct 30 23:42:49.323779 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:49.323684    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0779b720542efa369eebac415a07f5192a47a31647345a077ae0cc735bcb8bb3"
Oct 30 23:42:50.217353 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.217320    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"]
Oct 30 23:42:50.217929 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.217915    4555 cpu_manager.go:414] "containerMap: removing container" podUID="eba3b5bf-02e9-43a9-830e-6aecc3162d0c" containerName="revoke-keys"
Oct 30 23:42:50.217977 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.217932    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba3b5bf-02e9-43a9-830e-6aecc3162d0c" containerName="revoke-keys"
Oct 30 23:42:50.218072 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.218063    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba3b5bf-02e9-43a9-830e-6aecc3162d0c" containerName="revoke-keys"
Oct 30 23:42:50.221732 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.221714    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.229938 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.226448    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:42:50.229938 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.226528    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:42:50.232806 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.232779    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"]
Oct 30 23:42:50.374778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.374726    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414683df-56f9-4d37-af39-3eee805e1000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07e3845f-r2z9k\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.374978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.374900    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht9xt\" (UniqueName: \"kubernetes.io/projected/414683df-56f9-4d37-af39-3eee805e1000-kube-api-access-ht9xt\") pod \"maas-api-revoke-sub-07e3845f-r2z9k\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.374978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.374954    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414683df-56f9-4d37-af39-3eee805e1000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07e3845f-r2z9k\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.476828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.476628    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414683df-56f9-4d37-af39-3eee805e1000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07e3845f-r2z9k\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.476828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.476744    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht9xt\" (UniqueName: \"kubernetes.io/projected/414683df-56f9-4d37-af39-3eee805e1000-kube-api-access-ht9xt\") pod \"maas-api-revoke-sub-07e3845f-r2z9k\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.476828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.476818    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414683df-56f9-4d37-af39-3eee805e1000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07e3845f-r2z9k\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.477846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.477819    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414683df-56f9-4d37-af39-3eee805e1000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07e3845f-r2z9k\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.479839 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.479811    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414683df-56f9-4d37-af39-3eee805e1000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07e3845f-r2z9k\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.489961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.489932    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht9xt\" (UniqueName: \"kubernetes.io/projected/414683df-56f9-4d37-af39-3eee805e1000-kube-api-access-ht9xt\") pod \"maas-api-revoke-sub-07e3845f-r2z9k\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.541804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.541772    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:50.850324 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:42:50.846615    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414683df_56f9_4d37_af39_3eee805e1000.slice/crio-7ddaff54a51dc646d774e5933c80601e456c91dc82a16d19cbb685055cda5964 WatchSource:0}: Error finding container 7ddaff54a51dc646d774e5933c80601e456c91dc82a16d19cbb685055cda5964: Status 404 returned error can't find the container with id 7ddaff54a51dc646d774e5933c80601e456c91dc82a16d19cbb685055cda5964
Oct 30 23:42:50.853308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:50.853278    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"]
Oct 30 23:42:51.339997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:51.339954    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k" event={"ID":"414683df-56f9-4d37-af39-3eee805e1000","Type":"ContainerStarted","Data":"cfdb409ad63052d14743813ba985299d72e882bad5b031baa4f80d5e83edf38e"}
Oct 30 23:42:51.340522 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:51.340022    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k" event={"ID":"414683df-56f9-4d37-af39-3eee805e1000","Type":"ContainerStarted","Data":"7ddaff54a51dc646d774e5933c80601e456c91dc82a16d19cbb685055cda5964"}
Oct 30 23:42:51.364004 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:51.363953    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k" podStartSLOduration=1.363938032 podStartE2EDuration="1.363938032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:51.361165687 +0000 UTC m=+1430.287803270" watchObservedRunningTime="2026-10-30 23:42:51.363938032 +0000 UTC m=+1430.290575625"
Oct 30 23:42:52.349482 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:52.349445    4555 generic.go:356] "Generic (PLEG): container finished" podID="414683df-56f9-4d37-af39-3eee805e1000" containerID="cfdb409ad63052d14743813ba985299d72e882bad5b031baa4f80d5e83edf38e" exitCode=0
Oct 30 23:42:52.349906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:52.349535    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k" event={"ID":"414683df-56f9-4d37-af39-3eee805e1000","Type":"ContainerDied","Data":"cfdb409ad63052d14743813ba985299d72e882bad5b031baa4f80d5e83edf38e"}
Oct 30 23:42:53.645747 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:53.645723    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:53.733017 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:53.732974    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414683df-56f9-4d37-af39-3eee805e1000-kube-api-access-ht9xt\" (UniqueName: \"kubernetes.io/projected/414683df-56f9-4d37-af39-3eee805e1000-kube-api-access-ht9xt\") pod \"414683df-56f9-4d37-af39-3eee805e1000\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") "
Oct 30 23:42:53.733197 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:53.733104    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414683df-56f9-4d37-af39-3eee805e1000-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414683df-56f9-4d37-af39-3eee805e1000-maas-api-xks-ca\") pod \"414683df-56f9-4d37-af39-3eee805e1000\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") "
Oct 30 23:42:53.733325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:53.733308    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414683df-56f9-4d37-af39-3eee805e1000-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414683df-56f9-4d37-af39-3eee805e1000-maas-api-openshift-ca\") pod \"414683df-56f9-4d37-af39-3eee805e1000\" (UID: \"414683df-56f9-4d37-af39-3eee805e1000\") "
Oct 30 23:42:53.733679 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:53.733649    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414683df-56f9-4d37-af39-3eee805e1000-maas-api-openshift-ca" pod "414683df-56f9-4d37-af39-3eee805e1000" (UID: "414683df-56f9-4d37-af39-3eee805e1000"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:53.734019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:53.733998    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414683df-56f9-4d37-af39-3eee805e1000-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:53.735231 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:53.735202    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414683df-56f9-4d37-af39-3eee805e1000-kube-api-access-ht9xt" pod "414683df-56f9-4d37-af39-3eee805e1000" (UID: "414683df-56f9-4d37-af39-3eee805e1000"). InnerVolumeSpecName "kube-api-access-ht9xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:53.735324 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:53.735236    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414683df-56f9-4d37-af39-3eee805e1000-maas-api-xks-ca" pod "414683df-56f9-4d37-af39-3eee805e1000" (UID: "414683df-56f9-4d37-af39-3eee805e1000"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:53.835467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:53.835431    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht9xt\" (UniqueName: \"kubernetes.io/projected/414683df-56f9-4d37-af39-3eee805e1000-kube-api-access-ht9xt\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:53.835467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:53.835463    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414683df-56f9-4d37-af39-3eee805e1000-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:54.370014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:54.369978    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k" event={"ID":"414683df-56f9-4d37-af39-3eee805e1000","Type":"ContainerDied","Data":"7ddaff54a51dc646d774e5933c80601e456c91dc82a16d19cbb685055cda5964"}
Oct 30 23:42:54.370014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:54.370000    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"
Oct 30 23:42:54.370224 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:42:54.370041    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ddaff54a51dc646d774e5933c80601e456c91dc82a16d19cbb685055cda5964"
Oct 30 23:42:57.447855 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:42:57.447806    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:43:00.537823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.537786    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"]
Oct 30 23:43:00.538643 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.538622    4555 cpu_manager.go:414] "containerMap: removing container" podUID="414683df-56f9-4d37-af39-3eee805e1000" containerName="revoke-keys"
Oct 30 23:43:00.538772 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.538643    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414683df-56f9-4d37-af39-3eee805e1000" containerName="revoke-keys"
Oct 30 23:43:00.538831 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.538780    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="414683df-56f9-4d37-af39-3eee805e1000" containerName="revoke-keys"
Oct 30 23:43:00.543892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.543870    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:00.552962 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.552931    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:43:00.553201 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.553186    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:43:00.554652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.554452    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"]
Oct 30 23:43:00.640650 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.640613    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83b52579-dpp2n\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:00.640828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.640795    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83b52579-dpp2n\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:00.640828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.640818    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h464s\" (UniqueName: \"kubernetes.io/projected/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-kube-api-access-h464s\") pod \"maas-api-revoke-sub-83b52579-dpp2n\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:00.746416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.742750    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83b52579-dpp2n\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:00.746416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.743096    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83b52579-dpp2n\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:00.746416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.743143    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h464s\" (UniqueName: \"kubernetes.io/projected/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-kube-api-access-h464s\") pod \"maas-api-revoke-sub-83b52579-dpp2n\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:00.746416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.745763    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83b52579-dpp2n\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:00.758611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.758579    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83b52579-dpp2n\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:00.761950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.761905    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h464s\" (UniqueName: \"kubernetes.io/projected/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-kube-api-access-h464s\") pod \"maas-api-revoke-sub-83b52579-dpp2n\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:00.863628 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:00.863546    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:01.167951 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:43:01.167513    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2ee303_1b62_47cc_bc03_c1dce4cf2647.slice/crio-37968ae83dcb58abd623554bb5317e378bffa330d65eaba763a10d23de4e77b0 WatchSource:0}: Error finding container 37968ae83dcb58abd623554bb5317e378bffa330d65eaba763a10d23de4e77b0: Status 404 returned error can't find the container with id 37968ae83dcb58abd623554bb5317e378bffa330d65eaba763a10d23de4e77b0
Oct 30 23:43:01.176657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:01.175801    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"]
Oct 30 23:43:01.438177 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:01.438090    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n" event={"ID":"cf2ee303-1b62-47cc-bc03-c1dce4cf2647","Type":"ContainerStarted","Data":"37968ae83dcb58abd623554bb5317e378bffa330d65eaba763a10d23de4e77b0"}
Oct 30 23:43:02.448774 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:02.448740    4555 generic.go:356] "Generic (PLEG): container finished" podID="cf2ee303-1b62-47cc-bc03-c1dce4cf2647" containerID="8dc3435e08f3c807b806278f3e044e2890f129aacd57d1ee8b7846f8904ed465" exitCode=0
Oct 30 23:43:02.449185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:02.448837    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n" event={"ID":"cf2ee303-1b62-47cc-bc03-c1dce4cf2647","Type":"ContainerDied","Data":"8dc3435e08f3c807b806278f3e044e2890f129aacd57d1ee8b7846f8904ed465"}
Oct 30 23:43:03.693394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:03.688174    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:03.728793 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:03.728696    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-kube-api-access-h464s\" (UniqueName: \"kubernetes.io/projected/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-kube-api-access-h464s\") pod \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") "
Oct 30 23:43:03.728973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:03.728884    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-openshift-ca\") pod \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") "
Oct 30 23:43:03.728973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:03.728967    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-xks-ca\") pod \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\" (UID: \"cf2ee303-1b62-47cc-bc03-c1dce4cf2647\") "
Oct 30 23:43:03.729328 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:03.729296    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-openshift-ca" pod "cf2ee303-1b62-47cc-bc03-c1dce4cf2647" (UID: "cf2ee303-1b62-47cc-bc03-c1dce4cf2647"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:03.729855 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:03.729826    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:03.731075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:03.731046    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-kube-api-access-h464s" pod "cf2ee303-1b62-47cc-bc03-c1dce4cf2647" (UID: "cf2ee303-1b62-47cc-bc03-c1dce4cf2647"). InnerVolumeSpecName "kube-api-access-h464s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:03.732006 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:03.731978    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-xks-ca" pod "cf2ee303-1b62-47cc-bc03-c1dce4cf2647" (UID: "cf2ee303-1b62-47cc-bc03-c1dce4cf2647"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:03.831869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:03.831825    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h464s\" (UniqueName: \"kubernetes.io/projected/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-kube-api-access-h464s\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:03.831869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:03.831869    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ee303-1b62-47cc-bc03-c1dce4cf2647-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:04.468048 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:04.468000    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n" event={"ID":"cf2ee303-1b62-47cc-bc03-c1dce4cf2647","Type":"ContainerDied","Data":"37968ae83dcb58abd623554bb5317e378bffa330d65eaba763a10d23de4e77b0"}
Oct 30 23:43:04.468353 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:04.468329    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37968ae83dcb58abd623554bb5317e378bffa330d65eaba763a10d23de4e77b0"
Oct 30 23:43:04.468581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:04.468567    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"
Oct 30 23:43:07.796543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.796506    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"]
Oct 30 23:43:07.797108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.797094    4555 cpu_manager.go:414] "containerMap: removing container" podUID="cf2ee303-1b62-47cc-bc03-c1dce4cf2647" containerName="revoke-keys"
Oct 30 23:43:07.797151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.797109    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2ee303-1b62-47cc-bc03-c1dce4cf2647" containerName="revoke-keys"
Oct 30 23:43:07.797230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.797221    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2ee303-1b62-47cc-bc03-c1dce4cf2647" containerName="revoke-keys"
Oct 30 23:43:07.801126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.801107    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:07.814041 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.814010    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"]
Oct 30 23:43:07.832353 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.832318    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:43:07.832563 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.832454    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:43:07.910766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.910721    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs8sh\" (UniqueName: \"kubernetes.io/projected/dc83e902-4d5a-46cc-b0ac-724c397fd528-kube-api-access-zs8sh\") pod \"maas-api-revoke-sub-72992f85-9g58w\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:07.910956 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.910857    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72992f85-9g58w\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:07.911038 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:07.911022    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72992f85-9g58w\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:08.012923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:08.012890    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72992f85-9g58w\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:08.013119 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:08.013078    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72992f85-9g58w\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:08.013228 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:08.013213    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs8sh\" (UniqueName: \"kubernetes.io/projected/dc83e902-4d5a-46cc-b0ac-724c397fd528-kube-api-access-zs8sh\") pod \"maas-api-revoke-sub-72992f85-9g58w\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:08.013894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:08.013862    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72992f85-9g58w\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:08.015877 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:08.015850    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72992f85-9g58w\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:08.082234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:08.082150    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs8sh\" (UniqueName: \"kubernetes.io/projected/dc83e902-4d5a-46cc-b0ac-724c397fd528-kube-api-access-zs8sh\") pod \"maas-api-revoke-sub-72992f85-9g58w\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:08.119499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:08.119472    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:08.557714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:08.535058    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"]
Oct 30 23:43:08.622431 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:43:08.620441    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc83e902_4d5a_46cc_b0ac_724c397fd528.slice/crio-01b3c967279a808c355503d7aa226009eb0c845a81e02d7d583dae51bf48ac3a WatchSource:0}: Error finding container 01b3c967279a808c355503d7aa226009eb0c845a81e02d7d583dae51bf48ac3a: Status 404 returned error can't find the container with id 01b3c967279a808c355503d7aa226009eb0c845a81e02d7d583dae51bf48ac3a
Oct 30 23:43:08.693404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:08.693332    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:43:09.675572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:09.674395    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w" event={"ID":"dc83e902-4d5a-46cc-b0ac-724c397fd528","Type":"ContainerStarted","Data":"92987329f85ecf15d9587eed3c23aae2ee0bf6223f6b6e5364612c11dbd6a169"}
Oct 30 23:43:09.675572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:09.674468    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w" event={"ID":"dc83e902-4d5a-46cc-b0ac-724c397fd528","Type":"ContainerStarted","Data":"01b3c967279a808c355503d7aa226009eb0c845a81e02d7d583dae51bf48ac3a"}
Oct 30 23:43:09.749280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:09.747960    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w" podStartSLOduration=2.747934858 podStartE2EDuration="2.747934858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:09.73182429 +0000 UTC m=+1448.658461872" watchObservedRunningTime="2026-10-30 23:43:09.747934858 +0000 UTC m=+1448.674572479"
Oct 30 23:43:10.239627 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:10.216952    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"]
Oct 30 23:43:10.239627 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:10.217310    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy" containerID="cri-o://9f9b508ca6200605b0e24d661be564221909cb46a8e5e647c31a3c9332b69b44" gracePeriod=30
Oct 30 23:43:10.884394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:10.874041    4555 generic.go:356] "Generic (PLEG): container finished" podID="dc83e902-4d5a-46cc-b0ac-724c397fd528" containerID="92987329f85ecf15d9587eed3c23aae2ee0bf6223f6b6e5364612c11dbd6a169" exitCode=0
Oct 30 23:43:10.884394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:10.874191    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w" event={"ID":"dc83e902-4d5a-46cc-b0ac-724c397fd528","Type":"ContainerDied","Data":"92987329f85ecf15d9587eed3c23aae2ee0bf6223f6b6e5364612c11dbd6a169"}
Oct 30 23:43:10.993395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:10.993027    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:11.471396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:11.466460    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:11.640897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:11.640812    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 23:43:11.935906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:11.932720    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 30 23:43:11.948125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:11.948073    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:43:12.001257 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:12.001215    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:43:12.013294 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:12.013266    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:43:12.082494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:12.065452    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:43:12.082815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:12.065592    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 23:43:12.234122 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:12.234034    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:43:12.553432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:12.546855    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:12.553432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:12.547422    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:43:12.914454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:12.852465    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 30 23:43:12.992395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:12.977503    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:43:12.992395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:12.978123    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:43:13.155397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:13.154992    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:43:13.428590 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:13.428529    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:43:13.428920 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:13.428872    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:13.701152 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:13.700265    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:13.732537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:13.732494    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:13.999445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:13.979874    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:14.072736 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.072689    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:14.098960 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.084045    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:14.193497 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.174021    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:43:14.360846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.360798    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:14.362430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.362395    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:43:14.392016 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.391972    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:14.408154 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.392279    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:14.408550 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.406728    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:43:14.412483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.412442    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:14.678286 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.678250    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:14.737007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.736954    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:14.737948 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.737911    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:43:14.929334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.910778    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc83e902-4d5a-46cc-b0ac-724c397fd528-kube-api-access-zs8sh\" (UniqueName: \"kubernetes.io/projected/dc83e902-4d5a-46cc-b0ac-724c397fd528-kube-api-access-zs8sh\") pod \"dc83e902-4d5a-46cc-b0ac-724c397fd528\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") "
Oct 30 23:43:14.929334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.911062    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-xks-ca\") pod \"dc83e902-4d5a-46cc-b0ac-724c397fd528\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") "
Oct 30 23:43:14.929334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.911284    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-openshift-ca\") pod \"dc83e902-4d5a-46cc-b0ac-724c397fd528\" (UID: \"dc83e902-4d5a-46cc-b0ac-724c397fd528\") "
Oct 30 23:43:14.929651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.929563    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-openshift-ca" pod "dc83e902-4d5a-46cc-b0ac-724c397fd528" (UID: "dc83e902-4d5a-46cc-b0ac-724c397fd528"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:14.931658 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.931632    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:14.992826 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:14.992774    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:15.124079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.124021    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:43:15.126541 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.126512    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:43:15.225454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.208695    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:43:15.296398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.285043    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:15.445575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.445530    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:15.470447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.468920    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-xks-ca" pod "dc83e902-4d5a-46cc-b0ac-724c397fd528" (UID: "dc83e902-4d5a-46cc-b0ac-724c397fd528"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:15.515674 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.515571    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc83e902-4d5a-46cc-b0ac-724c397fd528-kube-api-access-zs8sh" pod "dc83e902-4d5a-46cc-b0ac-724c397fd528" (UID: "dc83e902-4d5a-46cc-b0ac-724c397fd528"). InnerVolumeSpecName "kube-api-access-zs8sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:15.632566 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.632528    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs8sh\" (UniqueName: \"kubernetes.io/projected/dc83e902-4d5a-46cc-b0ac-724c397fd528-kube-api-access-zs8sh\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:15.638861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.638832    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc83e902-4d5a-46cc-b0ac-724c397fd528-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:15.644496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.644472    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"
Oct 30 23:43:15.836408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.835722    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w" event={"ID":"dc83e902-4d5a-46cc-b0ac-724c397fd528","Type":"ContainerDied","Data":"01b3c967279a808c355503d7aa226009eb0c845a81e02d7d583dae51bf48ac3a"}
Oct 30 23:43:15.836408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.835908    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01b3c967279a808c355503d7aa226009eb0c845a81e02d7d583dae51bf48ac3a"
Oct 30 23:43:15.836408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.836015    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:43:15.836408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.836068    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:43:15.836408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.836113    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:43:15.881425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.874447    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 23:43:15.954390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:15.943810    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:16.099589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:16.099473    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:43:16.138399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:16.126201    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:43:16.267516 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:16.267469    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:16.281442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:16.277650    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:16.310461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:16.307479    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:16.310461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:16.307780    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:16.452496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:16.452453    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:16.887222 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:16.886664    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:16.905443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:16.905098    4555 generic.go:356] "Generic (PLEG): container finished" podID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerID="9f9b508ca6200605b0e24d661be564221909cb46a8e5e647c31a3c9332b69b44" exitCode=0
Oct 30 23:43:16.905443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:16.905352    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" event={"ID":"6f007baf-e820-44b9-b1ba-804cbd0fcbe3","Type":"ContainerDied","Data":"9f9b508ca6200605b0e24d661be564221909cb46a8e5e647c31a3c9332b69b44"}
Oct 30 23:43:17.111920 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.111071    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:17.153649 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.153518    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:17.303557 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.303505    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:17.304180 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.304149    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:43:17.307975 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.307949    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:43:17.341409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.309679    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:17.341990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.335132    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:17.342667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.342216    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:43:17.491455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.466468    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:43:17.878414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.864201    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:17.944496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:17.888710    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:18.144449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.134488    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:18.258830 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.221147    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:43:18.315394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.308482    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:43:18.362930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.348544    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:18.362930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.350884    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 23:43:18.362930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.359226    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:43:18.388182 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.387463    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:18.402048 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.400936    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:43:18.471195 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.454243    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:43:18.607158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.606107    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-podinfo\") pod \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") "
Oct 30 23:43:18.607158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.606229    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-envoy\") pod \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") "
Oct 30 23:43:18.607158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.606294    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-socket\") pod \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") "
Oct 30 23:43:18.607158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.606323    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-data\") pod \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") "
Oct 30 23:43:18.607158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.606425    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-credential-socket\") pod \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") "
Oct 30 23:43:18.607158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.606473    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-kube-api-access-rbgsp\" (UniqueName: \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-kube-api-access-rbgsp\") pod \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") "
Oct 30 23:43:18.607158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.606580    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-certs\") pod \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") "
Oct 30 23:43:18.607158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.606640    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-token\" (UniqueName: \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-token\") pod \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") "
Oct 30 23:43:18.607158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.606772    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istiod-ca-cert\") pod \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\" (UID: \"6f007baf-e820-44b9-b1ba-804cbd0fcbe3\") "
Oct 30 23:43:18.607847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.607697    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-credential-socket" pod "6f007baf-e820-44b9-b1ba-804cbd0fcbe3" (UID: "6f007baf-e820-44b9-b1ba-804cbd0fcbe3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:18.608876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.608262    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:18.629421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.627917    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-socket" pod "6f007baf-e820-44b9-b1ba-804cbd0fcbe3" (UID: "6f007baf-e820-44b9-b1ba-804cbd0fcbe3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:18.629421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.628472    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-data" pod "6f007baf-e820-44b9-b1ba-804cbd0fcbe3" (UID: "6f007baf-e820-44b9-b1ba-804cbd0fcbe3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:18.630304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.629897    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-certs" pod "6f007baf-e820-44b9-b1ba-804cbd0fcbe3" (UID: "6f007baf-e820-44b9-b1ba-804cbd0fcbe3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:18.630683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.630647    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istiod-ca-cert" pod "6f007baf-e820-44b9-b1ba-804cbd0fcbe3" (UID: "6f007baf-e820-44b9-b1ba-804cbd0fcbe3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:18.747002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.746824    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:18.747002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.746864    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:18.747002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.746890    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:18.747002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.746910    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:18.778788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.778731    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-kube-api-access-rbgsp" pod "6f007baf-e820-44b9-b1ba-804cbd0fcbe3" (UID: "6f007baf-e820-44b9-b1ba-804cbd0fcbe3"). InnerVolumeSpecName "kube-api-access-rbgsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:18.830845 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.810437    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:43:18.830845 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.811751    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-token" pod "6f007baf-e820-44b9-b1ba-804cbd0fcbe3" (UID: "6f007baf-e820-44b9-b1ba-804cbd0fcbe3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:18.830845 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.811845    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-envoy" pod "6f007baf-e820-44b9-b1ba-804cbd0fcbe3" (UID: "6f007baf-e820-44b9-b1ba-804cbd0fcbe3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:18.888761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.858958    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbgsp\" (UniqueName: \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-kube-api-access-rbgsp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:18.922985 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.922935    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:18.923302 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.923266    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:18.923508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.813931    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:43:18.926607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.926584    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:43:18.978845 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.978805    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-podinfo" pod "6f007baf-e820-44b9-b1ba-804cbd0fcbe3" (UID: "6f007baf-e820-44b9-b1ba-804cbd0fcbe3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:43:18.996395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.992263    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:43:19.024135 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:18.993888    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2jkbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:43:19.024356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:19.024336    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:19.024731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:19.024714    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:43:19.025682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:19.025659    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:43:19.029002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:19.028968    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f007baf-e820-44b9-b1ba-804cbd0fcbe3-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:19.171693 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:19.171664    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"
Oct 30 23:43:19.171851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:19.171662    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv" event={"ID":"6f007baf-e820-44b9-b1ba-804cbd0fcbe3","Type":"ContainerDied","Data":"5549daa0904dda13656d632aa151188b28a16646852b93e0d658b441939f77b1"}
Oct 30 23:43:19.171851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:19.171806    4555 scope.go:122] "RemoveContainer" containerID="9f9b508ca6200605b0e24d661be564221909cb46a8e5e647c31a3c9332b69b44"
Oct 30 23:43:19.232244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:19.232209    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"]
Oct 30 23:43:19.250926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:19.250894    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2jkbv"]
Oct 30 23:43:21.214785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:21.214746    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" path="/var/lib/kubelet/pods/6f007baf-e820-44b9-b1ba-804cbd0fcbe3/volumes"
Oct 30 23:43:26.670255 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.670193    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"]
Oct 30 23:43:26.672276 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.672248    4555 cpu_manager.go:414] "containerMap: removing container" podUID="dc83e902-4d5a-46cc-b0ac-724c397fd528" containerName="revoke-keys"
Oct 30 23:43:26.672477 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.672462    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc83e902-4d5a-46cc-b0ac-724c397fd528" containerName="revoke-keys"
Oct 30 23:43:26.672624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.672613    4555 cpu_manager.go:414] "containerMap: removing container" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy"
Oct 30 23:43:26.672700 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.672690    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy"
Oct 30 23:43:26.673054 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.673037    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f007baf-e820-44b9-b1ba-804cbd0fcbe3" containerName="istio-proxy"
Oct 30 23:43:26.673206 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.673189    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc83e902-4d5a-46cc-b0ac-724c397fd528" containerName="revoke-keys"
Oct 30 23:43:26.683684 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.683641    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.700300 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.700273    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"]
Oct 30 23:43:26.792034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.784204    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.793823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.793776    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.794152 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.794119    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.794410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.794364    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.797260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.797223    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25sv7\" (UniqueName: \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-kube-api-access-25sv7\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.797546 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.797512    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.797803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.797770    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.800204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.799544    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.800204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.799785    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.901472 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.901437    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.901644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.901537    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.901644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.901618    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.901788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.901648    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.901788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.901704    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.901788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.901734    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25sv7\" (UniqueName: \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-kube-api-access-25sv7\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.901926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.901802    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.901926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.901904    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.902039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.901931    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.902039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.901968    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.902146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.902051    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-credential-socket\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.902146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.902135    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-certs\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.902597 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.902568    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-data\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.902942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.902918    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istiod-ca-cert\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.905250 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.905228    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-envoy\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.905404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.905383    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-podinfo\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.918931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.918910    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25sv7\" (UniqueName: \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-kube-api-access-25sv7\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:26.919425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:26.919363    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-token\") pod \"e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") " pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:27.010296 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:27.010211    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:27.459397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:27.450626    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"]
Oct 30 23:43:27.487677 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:43:27.487521    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd4b677_566a_4c91_8f0f_c737e54ebf67.slice/crio-b745209323ef9a1a4ec98d0db0b9d3120ae85c257aa925b34a0df5e723c22b8b WatchSource:0}: Error finding container b745209323ef9a1a4ec98d0db0b9d3120ae85c257aa925b34a0df5e723c22b8b: Status 404 returned error can't find the container with id b745209323ef9a1a4ec98d0db0b9d3120ae85c257aa925b34a0df5e723c22b8b
Oct 30 23:43:27.506543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:27.492351    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:43:27.506543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:27.492506    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:43:27.506543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:27.492579    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:43:27.540492 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:43:27.540229    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:43:28.303256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:28.303215    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" event={"ID":"9dd4b677-566a-4c91-8f0f-c737e54ebf67","Type":"ContainerStarted","Data":"843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6"}
Oct 30 23:43:28.303736 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:28.303287    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" event={"ID":"9dd4b677-566a-4c91-8f0f-c737e54ebf67","Type":"ContainerStarted","Data":"b745209323ef9a1a4ec98d0db0b9d3120ae85c257aa925b34a0df5e723c22b8b"}
Oct 30 23:43:28.334295 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:28.334249    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" podStartSLOduration=2.334234197 podStartE2EDuration="2.334234197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:28.33082252 +0000 UTC m=+1467.257460105" watchObservedRunningTime="2026-10-30 23:43:28.334234197 +0000 UTC m=+1467.260871777"
Oct 30 23:43:29.149393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:29.145464    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:29.189800 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:29.189476    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 23:43:29.189800 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:29.189535    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 23:43:30.012256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:30.012209    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 23:43:30.012685 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:30.012290    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 23:43:31.000528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.000487    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"]
Oct 30 23:43:31.009101 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.006940    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.009353 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.009327    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"]
Oct 30 23:43:31.014335 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.012981    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 23:43:31.014335 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.013053    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 23:43:31.022849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.017586    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:43:31.022849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.017657    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:43:31.051023 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.050716    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e33d7f-dz7t5\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.051023 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.050806    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62rk4\" (UniqueName: \"kubernetes.io/projected/f309e1fd-b984-4023-9a90-031a688fd4a8-kube-api-access-62rk4\") pod \"maas-api-revoke-sub-82e33d7f-dz7t5\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.051023 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.050948    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e33d7f-dz7t5\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.158391 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.156452    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e33d7f-dz7t5\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.158391 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.156690    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e33d7f-dz7t5\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.158391 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.156789    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62rk4\" (UniqueName: \"kubernetes.io/projected/f309e1fd-b984-4023-9a90-031a688fd4a8-kube-api-access-62rk4\") pod \"maas-api-revoke-sub-82e33d7f-dz7t5\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.159705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.159673    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e33d7f-dz7t5\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.214396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.212637    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e33d7f-dz7t5\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.239445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.236819    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62rk4\" (UniqueName: \"kubernetes.io/projected/f309e1fd-b984-4023-9a90-031a688fd4a8-kube-api-access-62rk4\") pod \"maas-api-revoke-sub-82e33d7f-dz7t5\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.339428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.339355    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:31.843707 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:43:31.843449    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf309e1fd_b984_4023_9a90_031a688fd4a8.slice/crio-f3dbd04ae9f04ef538510dca79b5d0767342e7e62966c37d36c56b89239b2b0f WatchSource:0}: Error finding container f3dbd04ae9f04ef538510dca79b5d0767342e7e62966c37d36c56b89239b2b0f: Status 404 returned error can't find the container with id f3dbd04ae9f04ef538510dca79b5d0767342e7e62966c37d36c56b89239b2b0f
Oct 30 23:43:31.843707 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:31.843507    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"]
Oct 30 23:43:32.011631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:32.011600    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 23:43:32.011826 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:32.011656    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 23:43:32.596796 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:32.596757    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5" event={"ID":"f309e1fd-b984-4023-9a90-031a688fd4a8","Type":"ContainerStarted","Data":"73440c8e423ef469a165da6ed4f3128732b3086e20b2591c20f4ec210b135279"}
Oct 30 23:43:32.597186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:32.596859    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5" event={"ID":"f309e1fd-b984-4023-9a90-031a688fd4a8","Type":"ContainerStarted","Data":"f3dbd04ae9f04ef538510dca79b5d0767342e7e62966c37d36c56b89239b2b0f"}
Oct 30 23:43:32.694442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:32.689899    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5" podStartSLOduration=2.689875437 podStartE2EDuration="2.689875437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:32.667737564 +0000 UTC m=+1471.594375145" watchObservedRunningTime="2026-10-30 23:43:32.689875437 +0000 UTC m=+1471.616513019"
Oct 30 23:43:33.011105 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:33.011073    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 23:43:33.011219 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:33.011140    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 23:43:33.607540 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:33.607503    4555 generic.go:356] "Generic (PLEG): container finished" podID="f309e1fd-b984-4023-9a90-031a688fd4a8" containerID="73440c8e423ef469a165da6ed4f3128732b3086e20b2591c20f4ec210b135279" exitCode=0
Oct 30 23:43:33.607967 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:33.607601    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5" event={"ID":"f309e1fd-b984-4023-9a90-031a688fd4a8","Type":"ContainerDied","Data":"73440c8e423ef469a165da6ed4f3128732b3086e20b2591c20f4ec210b135279"}
Oct 30 23:43:34.011735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:34.011640    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 23:43:34.011735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:34.011694    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 23:43:34.744598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:34.744525    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"]
Oct 30 23:43:34.760686 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:34.760660    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:43:34.783663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:34.783609    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e4285d4a\""
Oct 30 23:43:34.804528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:34.798306    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"]
Oct 30 23:43:34.856715 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:34.856689    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:34.880447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:34.872796    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68\" (UID: \"4bdd66cc-8e82-403d-84e0-d0eaba1410c1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:43:34.880447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:34.872857    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvf42\" (UniqueName: \"kubernetes.io/projected/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-kube-api-access-dvf42\") pod \"payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68\" (UID: \"4bdd66cc-8e82-403d-84e0-d0eaba1410c1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:43:35.009400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.001015    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68\" (UID: \"4bdd66cc-8e82-403d-84e0-d0eaba1410c1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:43:35.009400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.001073    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvf42\" (UniqueName: \"kubernetes.io/projected/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-kube-api-access-dvf42\") pod \"payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68\" (UID: \"4bdd66cc-8e82-403d-84e0-d0eaba1410c1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:43:35.024849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.016541    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68\" (UID: \"4bdd66cc-8e82-403d-84e0-d0eaba1410c1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:43:35.029418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.022547    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 23:43:35.029418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.026539    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 23:43:35.097415 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.095436    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"]
Oct 30 23:43:35.097415 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.096473    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f309e1fd-b984-4023-9a90-031a688fd4a8" containerName="revoke-keys"
Oct 30 23:43:35.097415 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.096494    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f309e1fd-b984-4023-9a90-031a688fd4a8" containerName="revoke-keys"
Oct 30 23:43:35.097415 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.096650    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f309e1fd-b984-4023-9a90-031a688fd4a8" containerName="revoke-keys"
Oct 30 23:43:35.115573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.114175    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-openshift-ca\") pod \"f309e1fd-b984-4023-9a90-031a688fd4a8\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") "
Oct 30 23:43:35.115573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.114235    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f309e1fd-b984-4023-9a90-031a688fd4a8-kube-api-access-62rk4\" (UniqueName: \"kubernetes.io/projected/f309e1fd-b984-4023-9a90-031a688fd4a8-kube-api-access-62rk4\") pod \"f309e1fd-b984-4023-9a90-031a688fd4a8\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") "
Oct 30 23:43:35.115573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.114435    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-xks-ca\") pod \"f309e1fd-b984-4023-9a90-031a688fd4a8\" (UID: \"f309e1fd-b984-4023-9a90-031a688fd4a8\") "
Oct 30 23:43:35.118788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.118756    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-openshift-ca" pod "f309e1fd-b984-4023-9a90-031a688fd4a8" (UID: "f309e1fd-b984-4023-9a90-031a688fd4a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:35.122270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.122227    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f309e1fd-b984-4023-9a90-031a688fd4a8-kube-api-access-62rk4" pod "f309e1fd-b984-4023-9a90-031a688fd4a8" (UID: "f309e1fd-b984-4023-9a90-031a688fd4a8"). InnerVolumeSpecName "kube-api-access-62rk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:35.124593 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.124552    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-xks-ca" pod "f309e1fd-b984-4023-9a90-031a688fd4a8" (UID: "f309e1fd-b984-4023-9a90-031a688fd4a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:35.125785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.125435    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvf42\" (UniqueName: \"kubernetes.io/projected/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-kube-api-access-dvf42\") pod \"payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68\" (UID: \"4bdd66cc-8e82-403d-84e0-d0eaba1410c1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:43:35.139611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.138566    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"]
Oct 30 23:43:35.139611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.138676    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.152872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.151753    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:43:35.219044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.219012    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:35.219212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.219048    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f309e1fd-b984-4023-9a90-031a688fd4a8-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:35.219212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.219066    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62rk4\" (UniqueName: \"kubernetes.io/projected/f309e1fd-b984-4023-9a90-031a688fd4a8-kube-api-access-62rk4\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:35.338399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.326239    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j87z9\" (UniqueName: \"kubernetes.io/projected/77b70d20-8022-4933-9a6f-44651ff60c49-kube-api-access-j87z9\") pod \"maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.338399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.326295    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.338399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.326442    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.428343 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.428256    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.428521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.428499    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j87z9\" (UniqueName: \"kubernetes.io/projected/77b70d20-8022-4933-9a6f-44651ff60c49-kube-api-access-j87z9\") pod \"maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.428576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.428545    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.429224 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.429187    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.430825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.430808    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.442347 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.442321    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j87z9\" (UniqueName: \"kubernetes.io/projected/77b70d20-8022-4933-9a6f-44651ff60c49-kube-api-access-j87z9\") pod \"maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.476545 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:43:35.476503    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdd66cc_8e82_403d_84e0_d0eaba1410c1.slice/crio-bbe24d2e5c6cd2d9d19896d506431025d342787bbd8ef5bf23f218be007ef34c WatchSource:0}: Error finding container bbe24d2e5c6cd2d9d19896d506431025d342787bbd8ef5bf23f218be007ef34c: Status 404 returned error can't find the container with id bbe24d2e5c6cd2d9d19896d506431025d342787bbd8ef5bf23f218be007ef34c
Oct 30 23:43:35.477974 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.477939    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"]
Oct 30 23:43:35.509585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.509559    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:35.670889 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.670846    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5" event={"ID":"f309e1fd-b984-4023-9a90-031a688fd4a8","Type":"ContainerDied","Data":"f3dbd04ae9f04ef538510dca79b5d0767342e7e62966c37d36c56b89239b2b0f"}
Oct 30 23:43:35.671076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.670959    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3dbd04ae9f04ef538510dca79b5d0767342e7e62966c37d36c56b89239b2b0f"
Oct 30 23:43:35.671151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.671088    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"
Oct 30 23:43:35.679129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.678343    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68" event={"ID":"4bdd66cc-8e82-403d-84e0-d0eaba1410c1","Type":"ContainerStarted","Data":"79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424"}
Oct 30 23:43:35.679129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.678414    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68" event={"ID":"4bdd66cc-8e82-403d-84e0-d0eaba1410c1","Type":"ContainerStarted","Data":"bbe24d2e5c6cd2d9d19896d506431025d342787bbd8ef5bf23f218be007ef34c"}
Oct 30 23:43:35.679129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.679057    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:43:35.723762 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.723630    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68" podStartSLOduration=1.723612019 podStartE2EDuration="1.723612019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:35.718095766 +0000 UTC m=+1474.644733371" watchObservedRunningTime="2026-10-30 23:43:35.723612019 +0000 UTC m=+1474.650249601"
Oct 30 23:43:35.874485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:35.866427    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"]
Oct 30 23:43:36.155626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:36.154283    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:36.155626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:36.154765    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:36.157492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:36.157053    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 30 23:43:36.692782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:36.692688    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58" event={"ID":"77b70d20-8022-4933-9a6f-44651ff60c49","Type":"ContainerStarted","Data":"29bd34534c38f6233f180063516a6048e22d966cb2288ecad1f7a445751ba728"}
Oct 30 23:43:36.692782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:36.692753    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58" event={"ID":"77b70d20-8022-4933-9a6f-44651ff60c49","Type":"ContainerStarted","Data":"5831b7ee0b034044c9d351dc43ef4d15f0bc30e6cf279641f6d5d0d4277a2b6f"}
Oct 30 23:43:36.717542 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:36.717494    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58" podStartSLOduration=2.717479732 podStartE2EDuration="2.717479732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:36.714106769 +0000 UTC m=+1475.640744351" watchObservedRunningTime="2026-10-30 23:43:36.717479732 +0000 UTC m=+1475.644117313"
Oct 30 23:43:37.707357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:37.707320    4555 generic.go:356] "Generic (PLEG): container finished" podID="77b70d20-8022-4933-9a6f-44651ff60c49" containerID="29bd34534c38f6233f180063516a6048e22d966cb2288ecad1f7a445751ba728" exitCode=0
Oct 30 23:43:37.707882 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:37.707538    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58" event={"ID":"77b70d20-8022-4933-9a6f-44651ff60c49","Type":"ContainerDied","Data":"29bd34534c38f6233f180063516a6048e22d966cb2288ecad1f7a445751ba728"}
Oct 30 23:43:39.147834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.147805    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:39.270414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.267328    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-xks-ca\") pod \"77b70d20-8022-4933-9a6f-44651ff60c49\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") "
Oct 30 23:43:39.270414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.267404    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b70d20-8022-4933-9a6f-44651ff60c49-kube-api-access-j87z9\" (UniqueName: \"kubernetes.io/projected/77b70d20-8022-4933-9a6f-44651ff60c49-kube-api-access-j87z9\") pod \"77b70d20-8022-4933-9a6f-44651ff60c49\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") "
Oct 30 23:43:39.270414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.267457    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-openshift-ca\") pod \"77b70d20-8022-4933-9a6f-44651ff60c49\" (UID: \"77b70d20-8022-4933-9a6f-44651ff60c49\") "
Oct 30 23:43:39.270414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.267856    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-openshift-ca" pod "77b70d20-8022-4933-9a6f-44651ff60c49" (UID: "77b70d20-8022-4933-9a6f-44651ff60c49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:39.270414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.268555    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:39.281766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.273873    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-xks-ca" pod "77b70d20-8022-4933-9a6f-44651ff60c49" (UID: "77b70d20-8022-4933-9a6f-44651ff60c49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:39.281766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.281628    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b70d20-8022-4933-9a6f-44651ff60c49-kube-api-access-j87z9" pod "77b70d20-8022-4933-9a6f-44651ff60c49" (UID: "77b70d20-8022-4933-9a6f-44651ff60c49"). InnerVolumeSpecName "kube-api-access-j87z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:39.369931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.369844    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b70d20-8022-4933-9a6f-44651ff60c49-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:39.369931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.369875    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j87z9\" (UniqueName: \"kubernetes.io/projected/77b70d20-8022-4933-9a6f-44651ff60c49-kube-api-access-j87z9\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:39.804075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.804043    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"
Oct 30 23:43:39.804075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.804053    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58" event={"ID":"77b70d20-8022-4933-9a6f-44651ff60c49","Type":"ContainerDied","Data":"5831b7ee0b034044c9d351dc43ef4d15f0bc30e6cf279641f6d5d0d4277a2b6f"}
Oct 30 23:43:39.804332 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:39.804101    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5831b7ee0b034044c9d351dc43ef4d15f0bc30e6cf279641f6d5d0d4277a2b6f"
Oct 30 23:43:46.083541 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.083316    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"]
Oct 30 23:43:46.084346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.084322    4555 cpu_manager.go:414] "containerMap: removing container" podUID="77b70d20-8022-4933-9a6f-44651ff60c49" containerName="revoke-keys"
Oct 30 23:43:46.084581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.084565    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b70d20-8022-4933-9a6f-44651ff60c49" containerName="revoke-keys"
Oct 30 23:43:46.085079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.084903    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b70d20-8022-4933-9a6f-44651ff60c49" containerName="revoke-keys"
Oct 30 23:43:46.098078 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.098045    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"]
Oct 30 23:43:46.098251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.098177    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.105592 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.105566    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e4285d4a-serving-cert\""
Oct 30 23:43:46.212397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.209450    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-metrics-certs\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.212397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.209522    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-maas-api-tls\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.212397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.209575    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa56495d-f1b5-4900-ad82-dd095da29519-openshift-service-ca\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.212397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.209656    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lgz5\" (UniqueName: \"kubernetes.io/projected/aa56495d-f1b5-4900-ad82-dd095da29519-kube-api-access-2lgz5\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.312085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.312045    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-metrics-certs\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.312303 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.312127    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-maas-api-tls\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.312303 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.312174    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa56495d-f1b5-4900-ad82-dd095da29519-openshift-service-ca\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.312303 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.312254    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lgz5\" (UniqueName: \"kubernetes.io/projected/aa56495d-f1b5-4900-ad82-dd095da29519-kube-api-access-2lgz5\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.313691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.313426    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa56495d-f1b5-4900-ad82-dd095da29519-openshift-service-ca\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.324456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.323503    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-metrics-certs\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.326163 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.326136    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-maas-api-tls\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.336242 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.335147    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lgz5\" (UniqueName: \"kubernetes.io/projected/aa56495d-f1b5-4900-ad82-dd095da29519-kube-api-access-2lgz5\") pod \"maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.446200 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.443659    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:46.720504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:46.717745    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:43:47.106330 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:47.106281    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"]
Oct 30 23:43:47.937427 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:47.936193    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh" event={"ID":"aa56495d-f1b5-4900-ad82-dd095da29519","Type":"ContainerStarted","Data":"899c9ccca57232c39096badc621bbc593013c3e29ab1b81880800be20a66ba84"}
Oct 30 23:43:49.259080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:49.259035    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:49.364392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:49.363263    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh" event={"ID":"aa56495d-f1b5-4900-ad82-dd095da29519","Type":"ContainerStarted","Data":"fd97ddb7e83783dece90b3579356ce3f664cb17afd86c59059a9e5f4be55f608"}
Oct 30 23:43:49.364392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:49.363345    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:49.367460 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:49.364869    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:49.367460 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:49.366865    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:49.487402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:49.467501    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh" podStartSLOduration=2.9915652120000003 podStartE2EDuration="3.467479188s" totalImagesPullingTime="475.913976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:49.410916505 +0000 UTC m=+1488.337554089" watchObservedRunningTime="2026-10-30 23:43:49.467479188 +0000 UTC m=+1488.394116767"
Oct 30 23:43:49.908942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:49.900988    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:50.097395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.095829    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:50.286456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.277128    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:50.286456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.277741    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:43:50.322546 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.318353    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:50.398011 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.322801    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:43:50.403903 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.370516    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:43:50.408409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.395049    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:43:50.434617 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.434583    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:50.435090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.409495    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:43:50.824396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.821031    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:43:50.828583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.828458    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:50.999901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:50.999854    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 23:43:51.227436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:51.225539    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:51.278503 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:51.278457    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:51.484565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:51.484449    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:51.506525 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:51.504160    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:51.506525 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:51.505275    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:43:51.747560 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:51.747455    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:51.938576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:51.938522    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:52.001657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.001555    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:52.013533 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.013495    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:43:52.174398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.127456    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:52.236413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.236203    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:52.293456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.239507    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:43:52.322398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.287464    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:43:52.410403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.407759    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:52.417427 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.413619    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:43:52.516540 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.516491    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:52.536848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.536807    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:43:52.969591 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:52.969479    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:53.003136 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.002151    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:53.129885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.128278    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:43:53.183653 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.168506    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:53.183653 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.182692    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:43:53.300455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.299096    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:53.330395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.330090    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:43:53.430512 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.430461    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:53.430933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.430903    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:43:53.620344 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.620242    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:53.621501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.621459    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:43:53.724396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.710730    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 30 23:43:53.749431 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:53.739818    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:43:54.007717 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:54.005149    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:43:54.058341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:54.057266    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:43:54.094916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:54.071299    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:43:54.158995 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:54.157769    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:43:55.634009 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:55.633983    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:43:57.446752 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:43:57.446681    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:43:59.638479 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:59.638450    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"]
Oct 30 23:43:59.643453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:43:59.643427    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebbe654c-k7b58"]
Oct 30 23:44:01.209785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:01.209750    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b70d20-8022-4933-9a6f-44651ff60c49" path="/var/lib/kubelet/pods/77b70d20-8022-4933-9a6f-44651ff60c49/volumes"
Oct 30 23:44:10.110854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.110815    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"]
Oct 30 23:44:10.117343 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.117322    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.145736 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.145704    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"]
Oct 30 23:44:10.178784 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.178753    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"]
Oct 30 23:44:10.179749 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.179715    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.179901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.179785    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.179963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.179909    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.180019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.179969    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.180104 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.180083    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b24d54-b21b-4168-a520-24717ced83a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.180218 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.180201    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.180273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.180232    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.180325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.180276    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b24d54-b21b-4168-a520-24717ced83a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.180436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.180383    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2pp\" (UniqueName: \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-kube-api-access-ws2pp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.183553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.183534    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.230435 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.230403    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"]
Oct 30 23:44:10.281593 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.281557    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b24d54-b21b-4168-a520-24717ced83a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.281767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.281651    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.281767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.281702    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.281767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.281746    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.281925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.281859    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b24d54-b21b-4168-a520-24717ced83a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.281925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.281906    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc735d7-ab52-40b3-bece-087e83a5ee61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.282034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.281973    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.282034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282019    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2pp\" (UniqueName: \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-kube-api-access-ws2pp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.282137 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282117    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.282229 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282213    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.282299 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282251    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.282354 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282331    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.282437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282419    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.282572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282557    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b24d54-b21b-4168-a520-24717ced83a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.282612 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282564    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.282652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282622    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.282737 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282724    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.282772 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282753    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.282826 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282813    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.282867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282840    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh4zc\" (UniqueName: \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-kube-api-access-vh4zc\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.282917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.282865    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.283092 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.283074    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.283175 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.283155    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.285047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.285027    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b24d54-b21b-4168-a520-24717ced83a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.285123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.285042    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.298477 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.298448    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.298606 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.298585    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2pp\" (UniqueName: \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-kube-api-access-ws2pp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wnl2n\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.384917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.384818    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.384917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.384902    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc735d7-ab52-40b3-bece-087e83a5ee61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.384956    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.385013    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.385042    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.385076    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.385121    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.385190    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.385255    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh4zc\" (UniqueName: \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-kube-api-access-vh4zc\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.385322    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.385479    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.385622    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.385661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.385646    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.386512 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.386485    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc735d7-ab52-40b3-bece-087e83a5ee61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.387687 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.387664    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.387964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.387946    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.413283 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.413256    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.413658 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.413637    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh4zc\" (UniqueName: \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-kube-api-access-vh4zc\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flxph\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.432939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.432910    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:10.508677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.506259    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:10.740931 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:44:10.740898    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b24d54_b21b_4168_a520_24717ced83a7.slice/crio-ef4b83a67e12a2c87b0549045f9168c0550cda53dfb66ab78a547f441b735b2f WatchSource:0}: Error finding container ef4b83a67e12a2c87b0549045f9168c0550cda53dfb66ab78a547f441b735b2f: Status 404 returned error can't find the container with id ef4b83a67e12a2c87b0549045f9168c0550cda53dfb66ab78a547f441b735b2f
Oct 30 23:44:10.742999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.742969    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"]
Oct 30 23:44:10.764403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.745101    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:44:10.764403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.745227    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:44:10.764403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.745322    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:44:10.841351 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:44:10.841322    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc735d7_ab52_40b3_bece_087e83a5ee61.slice/crio-c1fb97d5664c634aff4fadec9a812c7d9c7c35f381d81a1d47713777fc108ad3 WatchSource:0}: Error finding container c1fb97d5664c634aff4fadec9a812c7d9c7c35f381d81a1d47713777fc108ad3: Status 404 returned error can't find the container with id c1fb97d5664c634aff4fadec9a812c7d9c7c35f381d81a1d47713777fc108ad3
Oct 30 23:44:10.843992 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.843961    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"]
Oct 30 23:44:10.845110 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.845071    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:44:10.845304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.845192    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:44:10.845304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:10.845237    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:44:11.350007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.349964    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" event={"ID":"7fc735d7-ab52-40b3-bece-087e83a5ee61","Type":"ContainerStarted","Data":"74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd"}
Oct 30 23:44:11.350600 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.350036    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" event={"ID":"7fc735d7-ab52-40b3-bece-087e83a5ee61","Type":"ContainerStarted","Data":"c1fb97d5664c634aff4fadec9a812c7d9c7c35f381d81a1d47713777fc108ad3"}
Oct 30 23:44:11.352434 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.352400    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" event={"ID":"16b24d54-b21b-4168-a520-24717ced83a7","Type":"ContainerStarted","Data":"53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93"}
Oct 30 23:44:11.352657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.352456    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" event={"ID":"16b24d54-b21b-4168-a520-24717ced83a7","Type":"ContainerStarted","Data":"ef4b83a67e12a2c87b0549045f9168c0550cda53dfb66ab78a547f441b735b2f"}
Oct 30 23:44:11.383635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.383577    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podStartSLOduration=1.383560327 podStartE2EDuration="1.383560327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:11.377252489 +0000 UTC m=+1510.303890077" watchObservedRunningTime="2026-10-30 23:44:11.383560327 +0000 UTC m=+1510.310197914"
Oct 30 23:44:11.433401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.424412    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podStartSLOduration=1.42438618 podStartE2EDuration="1.42438618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:11.415658535 +0000 UTC m=+1510.342296126" watchObservedRunningTime="2026-10-30 23:44:11.42438618 +0000 UTC m=+1510.351023805"
Oct 30 23:44:11.456393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.433978    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:11.456393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.443560    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:11.456393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.443622    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:11.520750 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.519923    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:11.523669 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.523638    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:44:11.523921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:11.523895    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:44:12.434280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:12.434245    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:12.434861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:12.434316    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:12.520398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:12.515611    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:44:12.520398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:12.515673    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:44:13.440127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:13.440094    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:13.440711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:13.440679    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:13.510912 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:13.510877    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:44:13.511170 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:13.511147    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:44:14.434073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:14.434039    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:14.434296 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:14.434089    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:14.508311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:14.508276    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:44:14.509355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:14.509325    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:44:15.434246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:15.434192    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:15.434591 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:15.434560    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:15.518881 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:15.518833    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:44:15.519506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:15.519462    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:44:16.445022 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:16.444980    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:16.445286 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:16.445261    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:16.534717 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:16.534673    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:44:16.615693 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:16.615637    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:44:17.450909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:17.450873    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:17.451357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:17.451327    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:17.525398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:17.521515    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:44:17.555275 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:17.555216    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:44:18.436395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:18.435749    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:18.436395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:18.435846    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:18.517131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:18.517091    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:44:18.517452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:18.517419    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:44:19.438243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:19.438209    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:19.438845 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:19.438812    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:19.507468 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:19.507425    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:44:19.507786 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:19.507738    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:44:20.433818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:20.433780    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:20.434042 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:20.433831    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:20.507733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:20.507689    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:44:20.508232 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:20.507761    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:44:21.433595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:21.433550    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:44:21.433753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:21.433626    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:44:21.922464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:21.922429    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"]
Oct 30 23:44:21.930757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:21.930708    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"]
Oct 30 23:44:21.930934 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:21.930838    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:44:21.951164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:21.951128    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ab09c434-dockercfg-bv5mw\""
Oct 30 23:44:21.979400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:21.977579    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ab09c434\""
Oct 30 23:44:22.036581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.036545    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-plugins-config-volume\") pod \"payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n\" (UID: \"1e232c52-99f0-4b8b-b67c-a5d97c57aa20\") " pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:44:22.037107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.037083    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbx5\" (UniqueName: \"kubernetes.io/projected/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-kube-api-access-4mbx5\") pod \"payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n\" (UID: \"1e232c52-99f0-4b8b-b67c-a5d97c57aa20\") " pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:44:22.138852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.138812    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-plugins-config-volume\") pod \"payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n\" (UID: \"1e232c52-99f0-4b8b-b67c-a5d97c57aa20\") " pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:44:22.139360 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.139338    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbx5\" (UniqueName: \"kubernetes.io/projected/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-kube-api-access-4mbx5\") pod \"payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n\" (UID: \"1e232c52-99f0-4b8b-b67c-a5d97c57aa20\") " pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:44:22.162124 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.162067    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-plugins-config-volume\") pod \"payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n\" (UID: \"1e232c52-99f0-4b8b-b67c-a5d97c57aa20\") " pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:44:22.211475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.209192    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbx5\" (UniqueName: \"kubernetes.io/projected/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-kube-api-access-4mbx5\") pod \"payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n\" (UID: \"1e232c52-99f0-4b8b-b67c-a5d97c57aa20\") " pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:44:22.258246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.258206    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:22.260141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.259770    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:22.260337 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.260056    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:44:22.281136 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.281033    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:44:22.995607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:22.995566    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"]
Oct 30 23:44:23.000493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:23.000465    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:23.004805 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:23.004785    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:23.010417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:23.005757    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:44:23.089445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:23.089397    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n" event={"ID":"1e232c52-99f0-4b8b-b67c-a5d97c57aa20","Type":"ContainerStarted","Data":"f48be38e485250af674bb2ac4392f725fea43c6991c9bdd89f1128db4757ff39"}
Oct 30 23:44:23.092347 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:23.092319    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"]
Oct 30 23:44:23.117120 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:23.117088    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4108876-g4j82"]
Oct 30 23:44:23.226510 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:23.226429    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd231773-fef2-4945-bcb9-e09329508200" path="/var/lib/kubelet/pods/fd231773-fef2-4945-bcb9-e09329508200/volumes"
Oct 30 23:44:24.098188 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:24.098149    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n" event={"ID":"1e232c52-99f0-4b8b-b67c-a5d97c57aa20","Type":"ContainerStarted","Data":"fbdaaa246737e0798cda6177e5eb968b72db8bc8e97baf01469476690d5d5d25"}
Oct 30 23:44:24.098709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:24.098251    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:44:24.126586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:24.126532    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n" podStartSLOduration=3.126517703 podStartE2EDuration="3.126517703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:24.121184678 +0000 UTC m=+1523.047822259" watchObservedRunningTime="2026-10-30 23:44:24.126517703 +0000 UTC m=+1523.053155309"
Oct 30 23:44:27.447002 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:44:27.446958    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:44:27.603954 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:27.603924    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:44:35.042675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:35.042645    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"]
Oct 30 23:44:35.053462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:35.053364    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-350338c9-gjkr6"]
Oct 30 23:44:35.113144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:35.113109    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:44:35.211012 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:35.210983    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ab930a-55b4-46ea-9497-40fa4a329602" path="/var/lib/kubelet/pods/10ab930a-55b4-46ea-9497-40fa4a329602/volumes"
Oct 30 23:44:37.027419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:37.027386    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"]
Oct 30 23:44:37.033504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:37.033478    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f64a45-jdhw6"]
Oct 30 23:44:37.239039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:37.239008    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c052a30-2727-4348-9dbd-5e938012993c" path="/var/lib/kubelet/pods/2c052a30-2727-4348-9dbd-5e938012993c/volumes"
Oct 30 23:44:45.035801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:45.035760    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"]
Oct 30 23:44:45.040206 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:45.040180    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504a4e52-2rv9n"]
Oct 30 23:44:45.210088 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:45.210057    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4526b5d-ba8d-410e-8f1c-717ea858c74e" path="/var/lib/kubelet/pods/a4526b5d-ba8d-410e-8f1c-717ea858c74e/volumes"
Oct 30 23:44:48.028635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:48.028590    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"]
Oct 30 23:44:48.033798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:48.033771    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4570f25a-7dcwj"]
Oct 30 23:44:49.210706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:49.210665    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c267043-cb73-4967-ae79-d913a7d52942" path="/var/lib/kubelet/pods/5c267043-cb73-4967-ae79-d913a7d52942/volumes"
Oct 30 23:44:50.047986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:50.047952    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"]
Oct 30 23:44:50.061709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:50.061676    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6124650-76hzr"]
Oct 30 23:44:51.290363 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:51.290331    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2005ffc-37a6-4430-b9ef-632d1dbb28d2" path="/var/lib/kubelet/pods/f2005ffc-37a6-4430-b9ef-632d1dbb28d2/volumes"
Oct 30 23:44:55.498533 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:44:55.498497    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"]
Oct 30 23:44:57.446689 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:44:57.446652    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:45:00.044306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.044264    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"]
Oct 30 23:45:00.094400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.092913    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef13daf-c6bgf"]
Oct 30 23:45:00.180449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.180405    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"]
Oct 30 23:45:00.185557 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.185533    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"
Oct 30 23:45:00.188425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.188401    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"]
Oct 30 23:45:00.193724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.193691    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"
Oct 30 23:45:00.208514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.208475    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:45:00.214654 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.214630    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"]
Oct 30 23:45:00.242703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.242669    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"]
Oct 30 23:45:00.265074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.265047    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"
Oct 30 23:45:00.267647 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.267623    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"]
Oct 30 23:45:00.283235 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.283204    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5tkm\" (UniqueName: \"kubernetes.io/projected/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-kube-api-access-b5tkm\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np\" (UID: \"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"
Oct 30 23:45:00.283549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.283529    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np\" (UID: \"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"
Oct 30 23:45:00.283834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.283817    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8fm\" (UniqueName: \"kubernetes.io/projected/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-kube-api-access-rj8fm\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj\" (UID: \"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"
Oct 30 23:45:00.283999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.283983    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj\" (UID: \"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"
Oct 30 23:45:00.291952 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.291929    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"]
Oct 30 23:45:00.385709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.385613    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvtx\" (UniqueName: \"kubernetes.io/projected/dceb02fa-043f-4d27-b966-55b5e1879918-kube-api-access-stvtx\") pod \"maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn\" (UID: \"dceb02fa-043f-4d27-b966-55b5e1879918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"
Oct 30 23:45:00.385709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.385656    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8fm\" (UniqueName: \"kubernetes.io/projected/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-kube-api-access-rj8fm\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj\" (UID: \"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"
Oct 30 23:45:00.385950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.385782    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj\" (UID: \"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"
Oct 30 23:45:00.385950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.385932    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5tkm\" (UniqueName: \"kubernetes.io/projected/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-kube-api-access-b5tkm\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np\" (UID: \"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"
Oct 30 23:45:00.386127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.386001    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dceb02fa-043f-4d27-b966-55b5e1879918-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn\" (UID: \"dceb02fa-043f-4d27-b966-55b5e1879918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"
Oct 30 23:45:00.386127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.386050    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np\" (UID: \"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"
Oct 30 23:45:00.386639 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.386617    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj\" (UID: \"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"
Oct 30 23:45:00.386875 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.386852    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np\" (UID: \"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"
Oct 30 23:45:00.398438 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.398414    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5tkm\" (UniqueName: \"kubernetes.io/projected/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-kube-api-access-b5tkm\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np\" (UID: \"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"
Oct 30 23:45:00.398575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.398454    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8fm\" (UniqueName: \"kubernetes.io/projected/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-kube-api-access-rj8fm\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj\" (UID: \"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"
Oct 30 23:45:00.487786 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.487746    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dceb02fa-043f-4d27-b966-55b5e1879918-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn\" (UID: \"dceb02fa-043f-4d27-b966-55b5e1879918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"
Oct 30 23:45:00.487982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.487962    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvtx\" (UniqueName: \"kubernetes.io/projected/dceb02fa-043f-4d27-b966-55b5e1879918-kube-api-access-stvtx\") pod \"maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn\" (UID: \"dceb02fa-043f-4d27-b966-55b5e1879918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"
Oct 30 23:45:00.488423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.488404    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dceb02fa-043f-4d27-b966-55b5e1879918-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn\" (UID: \"dceb02fa-043f-4d27-b966-55b5e1879918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"
Oct 30 23:45:00.498864 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.498808    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n" podUID="1e232c52-99f0-4b8b-b67c-a5d97c57aa20" containerName="payload-processing" containerID="cri-o://fbdaaa246737e0798cda6177e5eb968b72db8bc8e97baf01469476690d5d5d25" gracePeriod=25
Oct 30 23:45:00.503064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.503037    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvtx\" (UniqueName: \"kubernetes.io/projected/dceb02fa-043f-4d27-b966-55b5e1879918-kube-api-access-stvtx\") pod \"maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn\" (UID: \"dceb02fa-043f-4d27-b966-55b5e1879918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"
Oct 30 23:45:00.513365 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.513338    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"
Oct 30 23:45:00.528164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.527972    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"
Oct 30 23:45:00.589437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.588631    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"
Oct 30 23:45:00.701991 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.701450    4555 generic.go:356] "Generic (PLEG): container finished" podID="1e232c52-99f0-4b8b-b67c-a5d97c57aa20" containerID="fbdaaa246737e0798cda6177e5eb968b72db8bc8e97baf01469476690d5d5d25" exitCode=0
Oct 30 23:45:00.701991 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.701780    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n" event={"ID":"1e232c52-99f0-4b8b-b67c-a5d97c57aa20","Type":"ContainerDied","Data":"fbdaaa246737e0798cda6177e5eb968b72db8bc8e97baf01469476690d5d5d25"}
Oct 30 23:45:00.815144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.815109    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"]
Oct 30 23:45:00.823205 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.823175    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"]
Oct 30 23:45:00.824502 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:45:00.823332    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21f2c8e_b3a5_4c9b_a56a_2ead9526797d.slice/crio-e5a48c5d3341aedf8ee7e3340d4ad046d908cc1af863d64c13ab8c4eb1480c32 WatchSource:0}: Error finding container e5a48c5d3341aedf8ee7e3340d4ad046d908cc1af863d64c13ab8c4eb1480c32: Status 404 returned error can't find the container with id e5a48c5d3341aedf8ee7e3340d4ad046d908cc1af863d64c13ab8c4eb1480c32
Oct 30 23:45:00.879434 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.879411    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:45:00.926660 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:45:00.926619    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddceb02fa_043f_4d27_b966_55b5e1879918.slice/crio-46c00a724a3804f45e938114a5744997c8d981fa69753aec28b61ad76d773fea WatchSource:0}: Error finding container 46c00a724a3804f45e938114a5744997c8d981fa69753aec28b61ad76d773fea: Status 404 returned error can't find the container with id 46c00a724a3804f45e938114a5744997c8d981fa69753aec28b61ad76d773fea
Oct 30 23:45:00.928833 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:00.928804    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"]
Oct 30 23:45:01.001689 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.001655    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-kube-api-access-4mbx5\" (UniqueName: \"kubernetes.io/projected/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-kube-api-access-4mbx5\") pod \"1e232c52-99f0-4b8b-b67c-a5d97c57aa20\" (UID: \"1e232c52-99f0-4b8b-b67c-a5d97c57aa20\") "
Oct 30 23:45:01.001973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.001952    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-plugins-config-volume\") pod \"1e232c52-99f0-4b8b-b67c-a5d97c57aa20\" (UID: \"1e232c52-99f0-4b8b-b67c-a5d97c57aa20\") "
Oct 30 23:45:01.002421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.002389    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-plugins-config-volume" pod "1e232c52-99f0-4b8b-b67c-a5d97c57aa20" (UID: "1e232c52-99f0-4b8b-b67c-a5d97c57aa20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:01.002757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.002725    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.004263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.004239    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-kube-api-access-4mbx5" pod "1e232c52-99f0-4b8b-b67c-a5d97c57aa20" (UID: "1e232c52-99f0-4b8b-b67c-a5d97c57aa20"). InnerVolumeSpecName "kube-api-access-4mbx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:01.031769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.031745    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"]
Oct 30 23:45:01.037619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.037598    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732b2e61-7dx2d"]
Oct 30 23:45:01.108670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.108634    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbx5\" (UniqueName: \"kubernetes.io/projected/1e232c52-99f0-4b8b-b67c-a5d97c57aa20-kube-api-access-4mbx5\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.212967 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.212881    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740f8605-1c38-4d02-9ebd-1c4101193dbc" path="/var/lib/kubelet/pods/740f8605-1c38-4d02-9ebd-1c4101193dbc/volumes"
Oct 30 23:45:01.213417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.213352    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9837086d-b44c-4638-9a48-57d10b590c32" path="/var/lib/kubelet/pods/9837086d-b44c-4638-9a48-57d10b590c32/volumes"
Oct 30 23:45:01.718966 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.718940    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"
Oct 30 23:45:01.718966 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.718944    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n" event={"ID":"1e232c52-99f0-4b8b-b67c-a5d97c57aa20","Type":"ContainerDied","Data":"f48be38e485250af674bb2ac4392f725fea43c6991c9bdd89f1128db4757ff39"}
Oct 30 23:45:01.719222 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.719011    4555 scope.go:122] "RemoveContainer" containerID="fbdaaa246737e0798cda6177e5eb968b72db8bc8e97baf01469476690d5d5d25"
Oct 30 23:45:01.721064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.721007    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" event={"ID":"dceb02fa-043f-4d27-b966-55b5e1879918","Type":"ContainerStarted","Data":"3911cf9e66c12416002d30c7efa8cd175a919a66d3cbf567477f275c17f67211"}
Oct 30 23:45:01.721195 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.721062    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" event={"ID":"dceb02fa-043f-4d27-b966-55b5e1879918","Type":"ContainerStarted","Data":"46c00a724a3804f45e938114a5744997c8d981fa69753aec28b61ad76d773fea"}
Oct 30 23:45:01.723389 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.723343    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" event={"ID":"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d","Type":"ContainerStarted","Data":"5833c440a3c9adb4910ff731470a6c29881d5364086d99b4be8c28f07ae37802"}
Oct 30 23:45:01.723506 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.723429    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" event={"ID":"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d","Type":"ContainerStarted","Data":"e5a48c5d3341aedf8ee7e3340d4ad046d908cc1af863d64c13ab8c4eb1480c32"}
Oct 30 23:45:01.725215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.725192    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" event={"ID":"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473","Type":"ContainerStarted","Data":"9727fee7b88c3f7ad14089b2d409693044403af026aa7d7a0aecba99abf718f3"}
Oct 30 23:45:01.725326 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.725232    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" event={"ID":"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473","Type":"ContainerStarted","Data":"6a25dbf7d1a56cc42654a86b8daf9ebd5c2af21ee8ecc90bae8964220da4bee0"}
Oct 30 23:45:01.744024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.743996    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"]
Oct 30 23:45:01.752820 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.752797    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ab09c434-65d9d9687f-dcc4n"]
Oct 30 23:45:01.773746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.773687    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" podStartSLOduration=1.773668286 podStartE2EDuration="1.773668286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:01.766887771 +0000 UTC m=+1560.693525353" watchObservedRunningTime="2026-10-30 23:45:01.773668286 +0000 UTC m=+1560.700305870"
Oct 30 23:45:01.787555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.787512    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" podStartSLOduration=1.787497443 podStartE2EDuration="1.787497443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:01.784613285 +0000 UTC m=+1560.711250867" watchObservedRunningTime="2026-10-30 23:45:01.787497443 +0000 UTC m=+1560.714135024"
Oct 30 23:45:01.803282 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:01.803239    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" podStartSLOduration=1.803220641 podStartE2EDuration="1.803220641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:01.802014833 +0000 UTC m=+1560.728652417" watchObservedRunningTime="2026-10-30 23:45:01.803220641 +0000 UTC m=+1560.729858221"
Oct 30 23:45:02.757612 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:02.756094    4555 generic.go:356] "Generic (PLEG): container finished" podID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerID="9727fee7b88c3f7ad14089b2d409693044403af026aa7d7a0aecba99abf718f3" exitCode=60
Oct 30 23:45:02.757612 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:02.756395    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" event={"ID":"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473","Type":"ContainerDied","Data":"9727fee7b88c3f7ad14089b2d409693044403af026aa7d7a0aecba99abf718f3"}
Oct 30 23:45:02.757612 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:02.756967    4555 scope.go:122] "RemoveContainer" containerID="9727fee7b88c3f7ad14089b2d409693044403af026aa7d7a0aecba99abf718f3"
Oct 30 23:45:02.766002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:02.765719    4555 generic.go:356] "Generic (PLEG): container finished" podID="dceb02fa-043f-4d27-b966-55b5e1879918" containerID="3911cf9e66c12416002d30c7efa8cd175a919a66d3cbf567477f275c17f67211" exitCode=60
Oct 30 23:45:02.766302 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:02.766280    4555 scope.go:122] "RemoveContainer" containerID="3911cf9e66c12416002d30c7efa8cd175a919a66d3cbf567477f275c17f67211"
Oct 30 23:45:02.766947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:02.766888    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" event={"ID":"dceb02fa-043f-4d27-b966-55b5e1879918","Type":"ContainerDied","Data":"3911cf9e66c12416002d30c7efa8cd175a919a66d3cbf567477f275c17f67211"}
Oct 30 23:45:02.771642 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:02.771485    4555 generic.go:356] "Generic (PLEG): container finished" podID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerID="5833c440a3c9adb4910ff731470a6c29881d5364086d99b4be8c28f07ae37802" exitCode=60
Oct 30 23:45:02.771802 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:02.771788    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" event={"ID":"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d","Type":"ContainerDied","Data":"5833c440a3c9adb4910ff731470a6c29881d5364086d99b4be8c28f07ae37802"}
Oct 30 23:45:02.772191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:02.772178    4555 scope.go:122] "RemoveContainer" containerID="5833c440a3c9adb4910ff731470a6c29881d5364086d99b4be8c28f07ae37802"
Oct 30 23:45:03.433048 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:03.433015    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e232c52-99f0-4b8b-b67c-a5d97c57aa20" path="/var/lib/kubelet/pods/1e232c52-99f0-4b8b-b67c-a5d97c57aa20/volumes"
Oct 30 23:45:03.869899 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:03.869857    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" event={"ID":"dceb02fa-043f-4d27-b966-55b5e1879918","Type":"ContainerStarted","Data":"e60c2166c99625e37b8464a59a1121dc3081924ecbf9ad69cdcb82ed7c9aa573"}
Oct 30 23:45:03.894545 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:03.886405    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" event={"ID":"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d","Type":"ContainerStarted","Data":"b17b79826a8d15dd531ea9474bcac53e4d68ed95f74660252be156e43580c998"}
Oct 30 23:45:03.917394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:03.898391    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" event={"ID":"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473","Type":"ContainerStarted","Data":"98da30ded541a7754c44429838bd3d6ca47b518f13ef77a7f9d030508593dde1"}
Oct 30 23:45:04.944164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:04.944127    4555 generic.go:356] "Generic (PLEG): container finished" podID="dceb02fa-043f-4d27-b966-55b5e1879918" containerID="e60c2166c99625e37b8464a59a1121dc3081924ecbf9ad69cdcb82ed7c9aa573" exitCode=60
Oct 30 23:45:04.944881 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:04.944854    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" event={"ID":"dceb02fa-043f-4d27-b966-55b5e1879918","Type":"ContainerDied","Data":"e60c2166c99625e37b8464a59a1121dc3081924ecbf9ad69cdcb82ed7c9aa573"}
Oct 30 23:45:04.945039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:04.945025    4555 scope.go:122] "RemoveContainer" containerID="3911cf9e66c12416002d30c7efa8cd175a919a66d3cbf567477f275c17f67211"
Oct 30 23:45:04.945676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:04.945655    4555 scope.go:122] "RemoveContainer" containerID="e60c2166c99625e37b8464a59a1121dc3081924ecbf9ad69cdcb82ed7c9aa573"
Oct 30 23:45:04.946592 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:04.946566    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn_odh-ai-gateway-infra(dceb02fa-043f-4d27-b966-55b5e1879918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" podUID="dceb02fa-043f-4d27-b966-55b5e1879918"
Oct 30 23:45:04.964958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:04.963116    4555 generic.go:356] "Generic (PLEG): container finished" podID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerID="b17b79826a8d15dd531ea9474bcac53e4d68ed95f74660252be156e43580c998" exitCode=60
Oct 30 23:45:04.964958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:04.963258    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" event={"ID":"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d","Type":"ContainerDied","Data":"b17b79826a8d15dd531ea9474bcac53e4d68ed95f74660252be156e43580c998"}
Oct 30 23:45:04.964958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:04.963787    4555 scope.go:122] "RemoveContainer" containerID="b17b79826a8d15dd531ea9474bcac53e4d68ed95f74660252be156e43580c998"
Oct 30 23:45:04.964958 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:04.964226    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np_odh-ai-gateway-infra(f21f2c8e-b3a5-4c9b-a56a-2ead9526797d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d"
Oct 30 23:45:04.984944 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:04.982642    4555 generic.go:356] "Generic (PLEG): container finished" podID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerID="98da30ded541a7754c44429838bd3d6ca47b518f13ef77a7f9d030508593dde1" exitCode=60
Oct 30 23:45:04.984944 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:04.982730    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" event={"ID":"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473","Type":"ContainerDied","Data":"98da30ded541a7754c44429838bd3d6ca47b518f13ef77a7f9d030508593dde1"}
Oct 30 23:45:04.984944 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:04.983343    4555 scope.go:122] "RemoveContainer" containerID="98da30ded541a7754c44429838bd3d6ca47b518f13ef77a7f9d030508593dde1"
Oct 30 23:45:04.984944 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:04.983802    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj_odh-ai-gateway-infra(a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473"
Oct 30 23:45:05.191396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:05.184605    4555 scope.go:122] "RemoveContainer" containerID="5833c440a3c9adb4910ff731470a6c29881d5364086d99b4be8c28f07ae37802"
Oct 30 23:45:05.322722 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:05.322590    4555 scope.go:122] "RemoveContainer" containerID="9727fee7b88c3f7ad14089b2d409693044403af026aa7d7a0aecba99abf718f3"
Oct 30 23:45:06.030016 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:06.029984    4555 scope.go:122] "RemoveContainer" containerID="e60c2166c99625e37b8464a59a1121dc3081924ecbf9ad69cdcb82ed7c9aa573"
Oct 30 23:45:06.030989 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:06.030954    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn_odh-ai-gateway-infra(dceb02fa-043f-4d27-b966-55b5e1879918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" podUID="dceb02fa-043f-4d27-b966-55b5e1879918"
Oct 30 23:45:06.040602 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:06.040575    4555 scope.go:122] "RemoveContainer" containerID="b17b79826a8d15dd531ea9474bcac53e4d68ed95f74660252be156e43580c998"
Oct 30 23:45:06.041233 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:06.041205    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np_odh-ai-gateway-infra(f21f2c8e-b3a5-4c9b-a56a-2ead9526797d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d"
Oct 30 23:45:06.121973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:06.121943    4555 scope.go:122] "RemoveContainer" containerID="98da30ded541a7754c44429838bd3d6ca47b518f13ef77a7f9d030508593dde1"
Oct 30 23:45:06.123150 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:06.123114    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj_odh-ai-gateway-infra(a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473"
Oct 30 23:45:07.096398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:07.085060    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:45:07.314397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:07.307416    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:45:07.323866 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:07.321447    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:45:07.536428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:07.520449    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:45:12.729572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.729490    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"]
Oct 30 23:45:12.732072 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.730154    4555 cpu_manager.go:414] "containerMap: removing container" podUID="1e232c52-99f0-4b8b-b67c-a5d97c57aa20" containerName="payload-processing"
Oct 30 23:45:12.732072 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.730165    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e232c52-99f0-4b8b-b67c-a5d97c57aa20" containerName="payload-processing"
Oct 30 23:45:12.732072 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.730273    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e232c52-99f0-4b8b-b67c-a5d97c57aa20" containerName="payload-processing"
Oct 30 23:45:12.735767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.735737    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:45:12.744074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.744049    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-6c5084-dockercfg-hczbk\""
Oct 30 23:45:12.744320 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.744303    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-6c5084\""
Oct 30 23:45:12.749674 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.749651    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"]
Oct 30 23:45:12.759626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.759603    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"]
Oct 30 23:45:12.779598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.779570    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"]
Oct 30 23:45:12.779774 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.779675    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:45:12.810026 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.809991    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc377667-f4cb-420e-867b-35f94720b4cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p\" (UID: \"bc377667-f4cb-420e-867b-35f94720b4cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:45:12.810210 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.810047    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wjt\" (UniqueName: \"kubernetes.io/projected/bc377667-f4cb-420e-867b-35f94720b4cd-kube-api-access-c2wjt\") pod \"payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p\" (UID: \"bc377667-f4cb-420e-867b-35f94720b4cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:45:12.810280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.810255    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5181de-449e-444d-a038-3defcb855281-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs\" (UID: \"0b5181de-449e-444d-a038-3defcb855281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:45:12.810401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.810365    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sb49\" (UniqueName: \"kubernetes.io/projected/0b5181de-449e-444d-a038-3defcb855281-kube-api-access-2sb49\") pod \"payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs\" (UID: \"0b5181de-449e-444d-a038-3defcb855281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:45:12.937397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.921774    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc377667-f4cb-420e-867b-35f94720b4cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p\" (UID: \"bc377667-f4cb-420e-867b-35f94720b4cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:45:12.937397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.921872    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wjt\" (UniqueName: \"kubernetes.io/projected/bc377667-f4cb-420e-867b-35f94720b4cd-kube-api-access-c2wjt\") pod \"payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p\" (UID: \"bc377667-f4cb-420e-867b-35f94720b4cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:45:12.937397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.922066    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5181de-449e-444d-a038-3defcb855281-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs\" (UID: \"0b5181de-449e-444d-a038-3defcb855281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:45:12.937397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.923217    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb49\" (UniqueName: \"kubernetes.io/projected/0b5181de-449e-444d-a038-3defcb855281-kube-api-access-2sb49\") pod \"payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs\" (UID: \"0b5181de-449e-444d-a038-3defcb855281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:45:12.937397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.924666    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5181de-449e-444d-a038-3defcb855281-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs\" (UID: \"0b5181de-449e-444d-a038-3defcb855281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:45:12.937397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.925191    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc377667-f4cb-420e-867b-35f94720b4cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p\" (UID: \"bc377667-f4cb-420e-867b-35f94720b4cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:45:12.957103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.957063    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wjt\" (UniqueName: \"kubernetes.io/projected/bc377667-f4cb-420e-867b-35f94720b4cd-kube-api-access-c2wjt\") pod \"payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p\" (UID: \"bc377667-f4cb-420e-867b-35f94720b4cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:45:12.974478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:12.971103    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sb49\" (UniqueName: \"kubernetes.io/projected/0b5181de-449e-444d-a038-3defcb855281-kube-api-access-2sb49\") pod \"payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs\" (UID: \"0b5181de-449e-444d-a038-3defcb855281\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:45:13.073668 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:13.073636    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:45:13.095644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:13.095619    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:45:13.476925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:13.476894    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"]
Oct 30 23:45:13.477498 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:45:13.477471    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc377667_f4cb_420e_867b_35f94720b4cd.slice/crio-3c12428c7f90a8bce147dfe0f3eed0c3ada0d3101dfe79608294c0fa75a3c2ee WatchSource:0}: Error finding container 3c12428c7f90a8bce147dfe0f3eed0c3ada0d3101dfe79608294c0fa75a3c2ee: Status 404 returned error can't find the container with id 3c12428c7f90a8bce147dfe0f3eed0c3ada0d3101dfe79608294c0fa75a3c2ee
Oct 30 23:45:13.486551 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:45:13.486524    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5181de_449e_444d_a038_3defcb855281.slice/crio-f755cf3bb230b10e9442e7836f539ed5c4cf5169e94dc55a14b1f1ba599a71b4 WatchSource:0}: Error finding container f755cf3bb230b10e9442e7836f539ed5c4cf5169e94dc55a14b1f1ba599a71b4: Status 404 returned error can't find the container with id f755cf3bb230b10e9442e7836f539ed5c4cf5169e94dc55a14b1f1ba599a71b4
Oct 30 23:45:13.488982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:13.488952    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"]
Oct 30 23:45:14.316156 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:14.316120    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p" event={"ID":"bc377667-f4cb-420e-867b-35f94720b4cd","Type":"ContainerStarted","Data":"19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8"}
Oct 30 23:45:14.316876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:14.316187    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p" event={"ID":"bc377667-f4cb-420e-867b-35f94720b4cd","Type":"ContainerStarted","Data":"3c12428c7f90a8bce147dfe0f3eed0c3ada0d3101dfe79608294c0fa75a3c2ee"}
Oct 30 23:45:14.316876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:14.316231    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:45:14.318841 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:14.318816    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs" event={"ID":"0b5181de-449e-444d-a038-3defcb855281","Type":"ContainerStarted","Data":"8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9"}
Oct 30 23:45:14.318958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:14.318861    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs" event={"ID":"0b5181de-449e-444d-a038-3defcb855281","Type":"ContainerStarted","Data":"f755cf3bb230b10e9442e7836f539ed5c4cf5169e94dc55a14b1f1ba599a71b4"}
Oct 30 23:45:14.318958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:14.318899    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:45:14.337573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:14.337522    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p" podStartSLOduration=2.337508975 podStartE2EDuration="2.337508975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:14.335683037 +0000 UTC m=+1573.262320621" watchObservedRunningTime="2026-10-30 23:45:14.337508975 +0000 UTC m=+1573.264146555"
Oct 30 23:45:14.358030 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:14.357974    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs" podStartSLOduration=2.35796172 podStartE2EDuration="2.35796172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:14.353853365 +0000 UTC m=+1573.280490957" watchObservedRunningTime="2026-10-30 23:45:14.35796172 +0000 UTC m=+1573.284599301"
Oct 30 23:45:15.209284 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:15.209260    4555 scope.go:122] "RemoveContainer" containerID="e60c2166c99625e37b8464a59a1121dc3081924ecbf9ad69cdcb82ed7c9aa573"
Oct 30 23:45:15.209520 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:15.209434    4555 scope.go:122] "RemoveContainer" containerID="b17b79826a8d15dd531ea9474bcac53e4d68ed95f74660252be156e43580c998"
Oct 30 23:45:16.361470 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:16.358362    4555 generic.go:356] "Generic (PLEG): container finished" podID="dceb02fa-043f-4d27-b966-55b5e1879918" containerID="11731c406f03fa83513310540bca7c445e19edf8d01890c3c21c154ff8616d8a" exitCode=60
Oct 30 23:45:16.361470 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:16.358527    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" event={"ID":"dceb02fa-043f-4d27-b966-55b5e1879918","Type":"ContainerDied","Data":"11731c406f03fa83513310540bca7c445e19edf8d01890c3c21c154ff8616d8a"}
Oct 30 23:45:16.361470 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:16.358586    4555 scope.go:122] "RemoveContainer" containerID="e60c2166c99625e37b8464a59a1121dc3081924ecbf9ad69cdcb82ed7c9aa573"
Oct 30 23:45:16.361470 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:16.359100    4555 scope.go:122] "RemoveContainer" containerID="11731c406f03fa83513310540bca7c445e19edf8d01890c3c21c154ff8616d8a"
Oct 30 23:45:16.361470 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:16.359492    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn_odh-ai-gateway-infra(dceb02fa-043f-4d27-b966-55b5e1879918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" podUID="dceb02fa-043f-4d27-b966-55b5e1879918"
Oct 30 23:45:16.365251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:16.365186    4555 generic.go:356] "Generic (PLEG): container finished" podID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerID="656d685320f353d2f6e1675f644536fffeaeed16440bd305f9c8b851ff4fa852" exitCode=60
Oct 30 23:45:16.365251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:16.365194    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" event={"ID":"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d","Type":"ContainerDied","Data":"656d685320f353d2f6e1675f644536fffeaeed16440bd305f9c8b851ff4fa852"}
Oct 30 23:45:16.365827 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:16.365805    4555 scope.go:122] "RemoveContainer" containerID="656d685320f353d2f6e1675f644536fffeaeed16440bd305f9c8b851ff4fa852"
Oct 30 23:45:16.366169 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:16.366148    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np_odh-ai-gateway-infra(f21f2c8e-b3a5-4c9b-a56a-2ead9526797d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d"
Oct 30 23:45:16.408656 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:16.408631    4555 scope.go:122] "RemoveContainer" containerID="b17b79826a8d15dd531ea9474bcac53e4d68ed95f74660252be156e43580c998"
Oct 30 23:45:17.202991 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.201850    4555 scope.go:122] "RemoveContainer" containerID="98da30ded541a7754c44429838bd3d6ca47b518f13ef77a7f9d030508593dde1"
Oct 30 23:45:17.399252 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.399217    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"]
Oct 30 23:45:17.416735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.416698    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"]
Oct 30 23:45:17.606418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.603899    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"
Oct 30 23:45:17.680701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.680671    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"
Oct 30 23:45:17.744247 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.744160    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dceb02fa-043f-4d27-b966-55b5e1879918-kube-api-access-stvtx\" (UniqueName: \"kubernetes.io/projected/dceb02fa-043f-4d27-b966-55b5e1879918-kube-api-access-stvtx\") pod \"dceb02fa-043f-4d27-b966-55b5e1879918\" (UID: \"dceb02fa-043f-4d27-b966-55b5e1879918\") "
Oct 30 23:45:17.744435 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.744344    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dceb02fa-043f-4d27-b966-55b5e1879918-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dceb02fa-043f-4d27-b966-55b5e1879918-maas-api-service-ca\") pod \"dceb02fa-043f-4d27-b966-55b5e1879918\" (UID: \"dceb02fa-043f-4d27-b966-55b5e1879918\") "
Oct 30 23:45:17.745207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.744927    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dceb02fa-043f-4d27-b966-55b5e1879918-maas-api-service-ca" pod "dceb02fa-043f-4d27-b966-55b5e1879918" (UID: "dceb02fa-043f-4d27-b966-55b5e1879918"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:17.745207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.745178    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dceb02fa-043f-4d27-b966-55b5e1879918-maas-api-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:45:17.759582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.759542    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dceb02fa-043f-4d27-b966-55b5e1879918-kube-api-access-stvtx" pod "dceb02fa-043f-4d27-b966-55b5e1879918" (UID: "dceb02fa-043f-4d27-b966-55b5e1879918"). InnerVolumeSpecName "kube-api-access-stvtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:17.845701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.845670    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-maas-api-service-ca\") pod \"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d\" (UID: \"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d\") "
Oct 30 23:45:17.845912 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.845899    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-kube-api-access-b5tkm\" (UniqueName: \"kubernetes.io/projected/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-kube-api-access-b5tkm\") pod \"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d\" (UID: \"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d\") "
Oct 30 23:45:17.846073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.846047    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-maas-api-service-ca" pod "f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" (UID: "f21f2c8e-b3a5-4c9b-a56a-2ead9526797d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:17.846357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.846340    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvtx\" (UniqueName: \"kubernetes.io/projected/dceb02fa-043f-4d27-b966-55b5e1879918-kube-api-access-stvtx\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:45:17.846445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.846382    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-maas-api-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:45:17.848063 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.848043    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-kube-api-access-b5tkm" pod "f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" (UID: "f21f2c8e-b3a5-4c9b-a56a-2ead9526797d"). InnerVolumeSpecName "kube-api-access-b5tkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:17.955392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:17.948108    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5tkm\" (UniqueName: \"kubernetes.io/projected/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d-kube-api-access-b5tkm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:45:18.394527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.394197    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"
Oct 30 23:45:18.394817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.394241    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np" event={"ID":"f21f2c8e-b3a5-4c9b-a56a-2ead9526797d","Type":"ContainerDied","Data":"e5a48c5d3341aedf8ee7e3340d4ad046d908cc1af863d64c13ab8c4eb1480c32"}
Oct 30 23:45:18.395050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.395027    4555 scope.go:122] "RemoveContainer" containerID="656d685320f353d2f6e1675f644536fffeaeed16440bd305f9c8b851ff4fa852"
Oct 30 23:45:18.396577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.396332    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"
Oct 30 23:45:18.397702 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.397648    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn" event={"ID":"dceb02fa-043f-4d27-b966-55b5e1879918","Type":"ContainerDied","Data":"46c00a724a3804f45e938114a5744997c8d981fa69753aec28b61ad76d773fea"}
Oct 30 23:45:18.402995 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.402964    4555 generic.go:356] "Generic (PLEG): container finished" podID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerID="fe65e79630255a329e0ce5a4756e4ff9d0fc0698ef54ba9cbc80a6573a783ba4" exitCode=60
Oct 30 23:45:18.403408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.403019    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" event={"ID":"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473","Type":"ContainerDied","Data":"fe65e79630255a329e0ce5a4756e4ff9d0fc0698ef54ba9cbc80a6573a783ba4"}
Oct 30 23:45:18.403533 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.403513    4555 scope.go:122] "RemoveContainer" containerID="fe65e79630255a329e0ce5a4756e4ff9d0fc0698ef54ba9cbc80a6573a783ba4"
Oct 30 23:45:18.403891 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:18.403868    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj_odh-ai-gateway-infra(a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473"
Oct 30 23:45:18.412033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.411809    4555 scope.go:122] "RemoveContainer" containerID="11731c406f03fa83513310540bca7c445e19edf8d01890c3c21c154ff8616d8a"
Oct 30 23:45:18.423381 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.423351    4555 scope.go:122] "RemoveContainer" containerID="98da30ded541a7754c44429838bd3d6ca47b518f13ef77a7f9d030508593dde1"
Oct 30 23:45:18.456080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.456044    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"]
Oct 30 23:45:18.461537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.461509    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-611ef3-29890065-dzhtn"]
Oct 30 23:45:18.482264 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.482226    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"]
Oct 30 23:45:18.486769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:18.486745    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3dcca-29890065-zk9np"]
Oct 30 23:45:19.212114 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:19.212076    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dceb02fa-043f-4d27-b966-55b5e1879918" path="/var/lib/kubelet/pods/dceb02fa-043f-4d27-b966-55b5e1879918/volumes"
Oct 30 23:45:19.212808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:19.212788    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" path="/var/lib/kubelet/pods/f21f2c8e-b3a5-4c9b-a56a-2ead9526797d/volumes"
Oct 30 23:45:19.511191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:19.511094    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"]
Oct 30 23:45:19.860348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:19.859238    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"
Oct 30 23:45:20.042502 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.042460    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-kube-api-access-rj8fm\" (UniqueName: \"kubernetes.io/projected/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-kube-api-access-rj8fm\") pod \"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473\" (UID: \"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473\") "
Oct 30 23:45:20.043023 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.042999    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-maas-api-service-ca\") pod \"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473\" (UID: \"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473\") "
Oct 30 23:45:20.043485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.043449    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-maas-api-service-ca" pod "a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" (UID: "a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:20.043990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.043956    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-maas-api-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:45:20.045351 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.045323    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-kube-api-access-rj8fm" pod "a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" (UID: "a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473"). InnerVolumeSpecName "kube-api-access-rj8fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:20.145188 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.145104    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj8fm\" (UniqueName: \"kubernetes.io/projected/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473-kube-api-access-rj8fm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:45:20.488313 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.488218    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj" event={"ID":"a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473","Type":"ContainerDied","Data":"6a25dbf7d1a56cc42654a86b8daf9ebd5c2af21ee8ecc90bae8964220da4bee0"}
Oct 30 23:45:20.488313 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.488254    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"
Oct 30 23:45:20.488313 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.488288    4555 scope.go:122] "RemoveContainer" containerID="fe65e79630255a329e0ce5a4756e4ff9d0fc0698ef54ba9cbc80a6573a783ba4"
Oct 30 23:45:20.516777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.516736    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"]
Oct 30 23:45:20.523745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:20.523706    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890065-whnhj"]
Oct 30 23:45:21.209903 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:21.209867    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" path="/var/lib/kubelet/pods/a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473/volumes"
Oct 30 23:45:25.331671 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:25.331644    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:45:25.332087 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:25.331708    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:45:27.446156 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:27.446117    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:45:32.547909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.547874    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"]
Oct 30 23:45:32.548980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.548958    4555 cpu_manager.go:414] "containerMap: removing container" podUID="dceb02fa-043f-4d27-b966-55b5e1879918" containerName="cleanup"
Oct 30 23:45:32.549079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.548985    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dceb02fa-043f-4d27-b966-55b5e1879918" containerName="cleanup"
Oct 30 23:45:32.549079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549017    4555 cpu_manager.go:414] "containerMap: removing container" podUID="dceb02fa-043f-4d27-b966-55b5e1879918" containerName="cleanup"
Oct 30 23:45:32.549079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549026    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dceb02fa-043f-4d27-b966-55b5e1879918" containerName="cleanup"
Oct 30 23:45:32.549079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549051    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerName="cleanup"
Oct 30 23:45:32.549079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549059    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerName="cleanup"
Oct 30 23:45:32.549079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549076    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549084    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549104    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549111    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549129    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549136    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549150    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549156    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549176    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549182    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549331    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549342    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="dceb02fa-043f-4d27-b966-55b5e1879918" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549352    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerName="cleanup"
Oct 30 23:45:32.549392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549361    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="dceb02fa-043f-4d27-b966-55b5e1879918" containerName="cleanup"
Oct 30 23:45:32.550019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549402    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerName="cleanup"
Oct 30 23:45:32.550019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549413    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerName="cleanup"
Oct 30 23:45:32.550019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549421    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a8b50c-e1a4-4b98-bef1-dfa38e6d2473" containerName="cleanup"
Oct 30 23:45:32.550019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.549429    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21f2c8e-b3a5-4c9b-a56a-2ead9526797d" containerName="cleanup"
Oct 30 23:45:32.555174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.555152    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.583420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.560191    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-a3e876-openshift-default-dockercfg-ddspk\""
Oct 30 23:45:32.583420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.569713    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"]
Oct 30 23:45:32.668397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.664206    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-credential-socket\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.668397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.664265    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-certs\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.668397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.664385    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-envoy\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.668397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.664439    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-podinfo\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.668397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.664529    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-data\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.668397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.664560    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-socket\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.953456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.952625    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-data\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.953456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.952681    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-socket\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.953456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.952717    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-token\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.953456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.952746    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istiod-ca-cert\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.953456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.953066    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-credential-socket\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.953456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.953099    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-certs\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.953456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.953196    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6flf\" (UniqueName: \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-kube-api-access-x6flf\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.953456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.953267    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-envoy\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.953456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.953327    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-podinfo\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.982392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.962588    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-certs\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.982392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.962897    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-data\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.982392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.963177    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-socket\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:32.982392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:32.963455    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-credential-socket\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:33.009392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.005179    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-envoy\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:33.012148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.012113    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-podinfo\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:33.068771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.068724    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-token\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:33.069046 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.069026    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istiod-ca-cert\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:33.069558 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.069524    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6flf\" (UniqueName: \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-kube-api-access-x6flf\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:33.071212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.071189    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istiod-ca-cert\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:33.122406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.118412    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6flf\" (UniqueName: \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-kube-api-access-x6flf\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:33.122406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.118908    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-token\") pod \"e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") " pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:33.183410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.182657    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:33.507443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.503529    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:45:33.562450 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.561045    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"]
Oct 30 23:45:33.677284 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:45:33.677252    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2029c09_8186_46dd_bc4f_aab88bde8fd1.slice/crio-1b66c5692e2596260761f33a4d0ba1be067ae16689f064dfaa7e3bf805857d2c WatchSource:0}: Error finding container 1b66c5692e2596260761f33a4d0ba1be067ae16689f064dfaa7e3bf805857d2c: Status 404 returned error can't find the container with id 1b66c5692e2596260761f33a4d0ba1be067ae16689f064dfaa7e3bf805857d2c
Oct 30 23:45:33.682956 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.682910    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:45:33.683188 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.683167    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:45:33.683319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:33.683302    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:45:34.759393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:34.756183    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" event={"ID":"c2029c09-8186-46dd-bc4f-aab88bde8fd1","Type":"ContainerStarted","Data":"3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3"}
Oct 30 23:45:34.759393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:34.756257    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" event={"ID":"c2029c09-8186-46dd-bc4f-aab88bde8fd1","Type":"ContainerStarted","Data":"1b66c5692e2596260761f33a4d0ba1be067ae16689f064dfaa7e3bf805857d2c"}
Oct 30 23:45:34.843418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:34.840662    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" podStartSLOduration=2.840643285 podStartE2EDuration="2.840643285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:34.812065117 +0000 UTC m=+1593.738702718" watchObservedRunningTime="2026-10-30 23:45:34.840643285 +0000 UTC m=+1593.767280868"
Oct 30 23:45:35.183820 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:35.183783    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:35.189879 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:35.189851    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:35.765786 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:35.765756    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:35.766847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:35.766827    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:45:46.653348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.653306    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"]
Oct 30 23:45:46.653964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.653948    4555 cpu_manager.go:414] "containerMap: removing container" podUID="dceb02fa-043f-4d27-b966-55b5e1879918" containerName="cleanup"
Oct 30 23:45:46.653964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.653964    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dceb02fa-043f-4d27-b966-55b5e1879918" containerName="cleanup"
Oct 30 23:45:46.654171 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.654104    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="dceb02fa-043f-4d27-b966-55b5e1879918" containerName="cleanup"
Oct 30 23:45:46.658977 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.658951    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:45:46.678208 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.677665    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"]
Oct 30 23:45:46.723732 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.723699    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6a162557\""
Oct 30 23:45:46.723944 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.723753    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6a162557-dockercfg-59xxl\""
Oct 30 23:45:46.877526 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.877467    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be309a26-4bd4-4f00-927c-f9b0fb48ca31-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw\" (UID: \"be309a26-4bd4-4f00-927c-f9b0fb48ca31\") " pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:45:46.877807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.877786    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8vdp\" (UniqueName: \"kubernetes.io/projected/be309a26-4bd4-4f00-927c-f9b0fb48ca31-kube-api-access-n8vdp\") pod \"payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw\" (UID: \"be309a26-4bd4-4f00-927c-f9b0fb48ca31\") " pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:45:46.995442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.995181    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be309a26-4bd4-4f00-927c-f9b0fb48ca31-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw\" (UID: \"be309a26-4bd4-4f00-927c-f9b0fb48ca31\") " pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:45:46.995442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.995242    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8vdp\" (UniqueName: \"kubernetes.io/projected/be309a26-4bd4-4f00-927c-f9b0fb48ca31-kube-api-access-n8vdp\") pod \"payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw\" (UID: \"be309a26-4bd4-4f00-927c-f9b0fb48ca31\") " pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:45:47.010397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:46.996758    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be309a26-4bd4-4f00-927c-f9b0fb48ca31-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw\" (UID: \"be309a26-4bd4-4f00-927c-f9b0fb48ca31\") " pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:45:47.058105 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:47.058076    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8vdp\" (UniqueName: \"kubernetes.io/projected/be309a26-4bd4-4f00-927c-f9b0fb48ca31-kube-api-access-n8vdp\") pod \"payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw\" (UID: \"be309a26-4bd4-4f00-927c-f9b0fb48ca31\") " pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:45:47.284698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:47.284662    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:45:47.661302 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:47.657045    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"]
Oct 30 23:45:48.084851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:48.082483    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw" event={"ID":"be309a26-4bd4-4f00-927c-f9b0fb48ca31","Type":"ContainerStarted","Data":"cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62"}
Oct 30 23:45:48.084851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:48.082557    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw" event={"ID":"be309a26-4bd4-4f00-927c-f9b0fb48ca31","Type":"ContainerStarted","Data":"c94f7d159767f7e8ae0e3e6df20749aa7d8d31160b67b8038cdb9f4386d85b6a"}
Oct 30 23:45:48.084851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:48.083992    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:45:48.107881 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:48.107825    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw" podStartSLOduration=2.107807237 podStartE2EDuration="2.107807237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:48.102725932 +0000 UTC m=+1607.029363515" watchObservedRunningTime="2026-10-30 23:45:48.107807237 +0000 UTC m=+1607.034444818"
Oct 30 23:45:51.370552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:51.370495    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"]
Oct 30 23:45:56.382393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:56.372452    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68" podUID="4bdd66cc-8e82-403d-84e0-d0eaba1410c1" containerName="payload-pre-processing" containerID="cri-o://79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424" gracePeriod=25
Oct 30 23:45:56.729319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:56.729296    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:45:56.872449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:56.872416    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-kube-api-access-dvf42\" (UniqueName: \"kubernetes.io/projected/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-kube-api-access-dvf42\") pod \"4bdd66cc-8e82-403d-84e0-d0eaba1410c1\" (UID: \"4bdd66cc-8e82-403d-84e0-d0eaba1410c1\") "
Oct 30 23:45:56.872650 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:56.872508    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-plugins-config-volume\") pod \"4bdd66cc-8e82-403d-84e0-d0eaba1410c1\" (UID: \"4bdd66cc-8e82-403d-84e0-d0eaba1410c1\") "
Oct 30 23:45:56.873476 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:56.873251    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-plugins-config-volume" pod "4bdd66cc-8e82-403d-84e0-d0eaba1410c1" (UID: "4bdd66cc-8e82-403d-84e0-d0eaba1410c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:56.874909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:56.874885    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-kube-api-access-dvf42" pod "4bdd66cc-8e82-403d-84e0-d0eaba1410c1" (UID: "4bdd66cc-8e82-403d-84e0-d0eaba1410c1"). InnerVolumeSpecName "kube-api-access-dvf42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:56.975518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:56.975428    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvf42\" (UniqueName: \"kubernetes.io/projected/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-kube-api-access-dvf42\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:45:56.975518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:56.975468    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdd66cc-8e82-403d-84e0-d0eaba1410c1-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:45:57.246129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:57.246025    4555 generic.go:356] "Generic (PLEG): container finished" podID="4bdd66cc-8e82-403d-84e0-d0eaba1410c1" containerID="79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424" exitCode=0
Oct 30 23:45:57.246129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:57.246115    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"
Oct 30 23:45:57.246384 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:57.246114    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68" event={"ID":"4bdd66cc-8e82-403d-84e0-d0eaba1410c1","Type":"ContainerDied","Data":"79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424"}
Oct 30 23:45:57.246384 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:57.246203    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68" event={"ID":"4bdd66cc-8e82-403d-84e0-d0eaba1410c1","Type":"ContainerDied","Data":"bbe24d2e5c6cd2d9d19896d506431025d342787bbd8ef5bf23f218be007ef34c"}
Oct 30 23:45:57.246384 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:57.246244    4555 scope.go:122] "RemoveContainer" containerID="79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424"
Oct 30 23:45:57.263625 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:57.263600    4555 scope.go:122] "RemoveContainer" containerID="79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424"
Oct 30 23:45:57.264046 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:57.264021    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424\": container with ID starting with 79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424 not found: ID does not exist" containerID="79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424"
Oct 30 23:45:57.264132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:57.264059    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424"} err="failed to get container status \"79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424\": rpc error: code = NotFound desc = could not find container \"79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424\": container with ID starting with 79ec2377c35378d7e82e2d286af15b53850f3524e57dfb22e6d2ce33476e8424 not found: ID does not exist"
Oct 30 23:45:57.280878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:57.280838    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"]
Oct 30 23:45:57.288682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:57.288656    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68"]
Oct 30 23:45:57.446128 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:45:57.446096    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:45:57.694402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:57.694310    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-e4285d4a-bd97d5975-z2q68" podUID="4bdd66cc-8e82-403d-84e0-d0eaba1410c1" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.102:50052\" within 1s: context deadline exceeded"
Oct 30 23:45:57.694582 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:45:57.694421    4555 logging.go:55] [core] [Channel #539 SubChannel #540]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.102:50052", ServerName: "10.133.0.102:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.102:50052: operation was canceled"
Oct 30 23:45:59.026698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:59.026663    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"]
Oct 30 23:45:59.027174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:59.026955    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh" podUID="aa56495d-f1b5-4900-ad82-dd095da29519" containerName="maas-api" containerID="cri-o://fd97ddb7e83783dece90b3579356ce3f664cb17afd86c59059a9e5f4be55f608" gracePeriod=30
Oct 30 23:45:59.217423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:59.214567    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bdd66cc-8e82-403d-84e0-d0eaba1410c1" path="/var/lib/kubelet/pods/4bdd66cc-8e82-403d-84e0-d0eaba1410c1/volumes"
Oct 30 23:45:59.506490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:59.462396    4555 generic.go:356] "Generic (PLEG): container finished" podID="aa56495d-f1b5-4900-ad82-dd095da29519" containerID="fd97ddb7e83783dece90b3579356ce3f664cb17afd86c59059a9e5f4be55f608" exitCode=0
Oct 30 23:45:59.506490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:59.462636    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh" event={"ID":"aa56495d-f1b5-4900-ad82-dd095da29519","Type":"ContainerDied","Data":"fd97ddb7e83783dece90b3579356ce3f664cb17afd86c59059a9e5f4be55f608"}
Oct 30 23:45:59.896260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:45:59.896228    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:46:00.044589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.044560    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-metrics-certs\") pod \"aa56495d-f1b5-4900-ad82-dd095da29519\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") "
Oct 30 23:46:00.045064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.044600    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa56495d-f1b5-4900-ad82-dd095da29519-kube-api-access-2lgz5\" (UniqueName: \"kubernetes.io/projected/aa56495d-f1b5-4900-ad82-dd095da29519-kube-api-access-2lgz5\") pod \"aa56495d-f1b5-4900-ad82-dd095da29519\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") "
Oct 30 23:46:00.045064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.044768    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-maas-api-tls\") pod \"aa56495d-f1b5-4900-ad82-dd095da29519\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") "
Oct 30 23:46:00.045064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.044894    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa56495d-f1b5-4900-ad82-dd095da29519-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa56495d-f1b5-4900-ad82-dd095da29519-openshift-service-ca\") pod \"aa56495d-f1b5-4900-ad82-dd095da29519\" (UID: \"aa56495d-f1b5-4900-ad82-dd095da29519\") "
Oct 30 23:46:00.045405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.045324    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa56495d-f1b5-4900-ad82-dd095da29519-openshift-service-ca" pod "aa56495d-f1b5-4900-ad82-dd095da29519" (UID: "aa56495d-f1b5-4900-ad82-dd095da29519"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:46:00.045697 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.045678    4555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa56495d-f1b5-4900-ad82-dd095da29519-openshift-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:00.047033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.047006    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa56495d-f1b5-4900-ad82-dd095da29519-kube-api-access-2lgz5" pod "aa56495d-f1b5-4900-ad82-dd095da29519" (UID: "aa56495d-f1b5-4900-ad82-dd095da29519"). InnerVolumeSpecName "kube-api-access-2lgz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:00.047033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.047020    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-metrics-certs" pod "aa56495d-f1b5-4900-ad82-dd095da29519" (UID: "aa56495d-f1b5-4900-ad82-dd095da29519"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:46:00.047187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.047104    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-maas-api-tls" pod "aa56495d-f1b5-4900-ad82-dd095da29519" (UID: "aa56495d-f1b5-4900-ad82-dd095da29519"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:46:00.104034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.104004    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:46:00.146959 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.146884    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-maas-api-tls\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:00.146959 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.146915    4555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa56495d-f1b5-4900-ad82-dd095da29519-metrics-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:00.146959 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.146929    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lgz5\" (UniqueName: \"kubernetes.io/projected/aa56495d-f1b5-4900-ad82-dd095da29519-kube-api-access-2lgz5\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:00.474585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.474489    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"
Oct 30 23:46:00.474585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.474519    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh" event={"ID":"aa56495d-f1b5-4900-ad82-dd095da29519","Type":"ContainerDied","Data":"899c9ccca57232c39096badc621bbc593013c3e29ab1b81880800be20a66ba84"}
Oct 30 23:46:00.474840 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.474596    4555 scope.go:122] "RemoveContainer" containerID="fd97ddb7e83783dece90b3579356ce3f664cb17afd86c59059a9e5f4be55f608"
Oct 30 23:46:00.505887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.505855    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"]
Oct 30 23:46:00.511268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:00.511240    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4285d4a-6b849bbc87-jwksh"]
Oct 30 23:46:01.216157 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:01.216116    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa56495d-f1b5-4900-ad82-dd095da29519" path="/var/lib/kubelet/pods/aa56495d-f1b5-4900-ad82-dd095da29519/volumes"
Oct 30 23:46:02.281441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:02.281404    4555 scope.go:122] "RemoveContainer" containerID="8d4e78ed36f7d3107bee6bcb0f4d42486735f4c67f408ab0b55206bee913339e"
Oct 30 23:46:02.293207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:02.293175    4555 scope.go:122] "RemoveContainer" containerID="23b90a949d4d4917dff9a66db01353c1a9eb314db66ac524e7d3a05a8068cb12"
Oct 30 23:46:02.310561 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:02.310534    4555 scope.go:122] "RemoveContainer" containerID="04150b5bdb1e6d97a65df1526cc8080db147585cb838cfa4b4bc120d39acb59d"
Oct 30 23:46:02.322574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:02.322429    4555 scope.go:122] "RemoveContainer" containerID="535425922ea5f5935f1b9309539ad48a50dce480e481e88531c98ef9883cd6c0"
Oct 30 23:46:02.333301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:02.333280    4555 scope.go:122] "RemoveContainer" containerID="20b7b4f9f260671f846b3d8e13608bd6ea4ac6649d8833569e73ef05a092dc91"
Oct 30 23:46:02.344141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:02.344123    4555 scope.go:122] "RemoveContainer" containerID="04c3c249dcda8d2948eb2573d6ae769309b34ed92b8a8ea7fbebf7df03c54577"
Oct 30 23:46:02.353522 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:02.353503    4555 scope.go:122] "RemoveContainer" containerID="fb702a8ff7660d802618be92731febb0aa444b1d47ed610d1e68556e94a895d4"
Oct 30 23:46:02.363466 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:02.363445    4555 scope.go:122] "RemoveContainer" containerID="f39e90beb935d35e872998360c4bf57bf33d9d4c26761a6fef6efdb98882f62b"
Oct 30 23:46:04.294677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:04.294638    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:46:07.022420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.022389    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"]
Oct 30 23:46:07.023207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.023186    4555 cpu_manager.go:414] "containerMap: removing container" podUID="4bdd66cc-8e82-403d-84e0-d0eaba1410c1" containerName="payload-pre-processing"
Oct 30 23:46:07.023322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.023209    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bdd66cc-8e82-403d-84e0-d0eaba1410c1" containerName="payload-pre-processing"
Oct 30 23:46:07.023322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.023246    4555 cpu_manager.go:414] "containerMap: removing container" podUID="aa56495d-f1b5-4900-ad82-dd095da29519" containerName="maas-api"
Oct 30 23:46:07.023322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.023255    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa56495d-f1b5-4900-ad82-dd095da29519" containerName="maas-api"
Oct 30 23:46:07.023530 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.023483    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bdd66cc-8e82-403d-84e0-d0eaba1410c1" containerName="payload-pre-processing"
Oct 30 23:46:07.023530 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.023504    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa56495d-f1b5-4900-ad82-dd095da29519" containerName="maas-api"
Oct 30 23:46:07.028839 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.028816    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.035812 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.035791    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"]
Oct 30 23:46:07.093451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.093151    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:46:07.093451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.093404    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:46:07.130214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.130170    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e2ee664-bwxws\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.130412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.130353    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdnfp\" (UniqueName: \"kubernetes.io/projected/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-kube-api-access-jdnfp\") pod \"maas-api-revoke-sub-2e2ee664-bwxws\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.130545 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.130527    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e2ee664-bwxws\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.238198 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.238155    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e2ee664-bwxws\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.238354 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.238337    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e2ee664-bwxws\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.238567 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.238548    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdnfp\" (UniqueName: \"kubernetes.io/projected/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-kube-api-access-jdnfp\") pod \"maas-api-revoke-sub-2e2ee664-bwxws\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.250837 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.250796    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e2ee664-bwxws\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.252761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.252695    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e2ee664-bwxws\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.257474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.257449    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdnfp\" (UniqueName: \"kubernetes.io/projected/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-kube-api-access-jdnfp\") pod \"maas-api-revoke-sub-2e2ee664-bwxws\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.346789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.346690    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:07.814395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:07.811977    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"]
Oct 30 23:46:08.052228 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.051519    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"]
Oct 30 23:46:08.057411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.057351    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.083401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.081323    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"]
Oct 30 23:46:08.091994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.091707    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ca1003-n6vfl\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.091994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.091863    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqd6\" (UniqueName: \"kubernetes.io/projected/24982555-2ef8-4285-8db9-d9b19f8ef63e-kube-api-access-9wqd6\") pod \"maas-api-revoke-sub-40ca1003-n6vfl\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.091994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.091893    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ca1003-n6vfl\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.194720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.194346    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqd6\" (UniqueName: \"kubernetes.io/projected/24982555-2ef8-4285-8db9-d9b19f8ef63e-kube-api-access-9wqd6\") pod \"maas-api-revoke-sub-40ca1003-n6vfl\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.194720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.194447    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ca1003-n6vfl\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.194967 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.194758    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ca1003-n6vfl\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.195461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.195440    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ca1003-n6vfl\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.294731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.292479    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqd6\" (UniqueName: \"kubernetes.io/projected/24982555-2ef8-4285-8db9-d9b19f8ef63e-kube-api-access-9wqd6\") pod \"maas-api-revoke-sub-40ca1003-n6vfl\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.294731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.293753    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ca1003-n6vfl\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.411281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.411237    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:08.779407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.774362    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws" event={"ID":"acec78f6-135c-4d2e-80e2-f2b70d70eb5c","Type":"ContainerStarted","Data":"3323d1d0afc2d16db881bd3f4766a294d909dc27d9b6119cb0c7f5f84ff28487"}
Oct 30 23:46:08.779407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.774452    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws" event={"ID":"acec78f6-135c-4d2e-80e2-f2b70d70eb5c","Type":"ContainerStarted","Data":"52105475ad90a8e390688d6ae6ef53f4264ffd2b8e6a5049200c49e251f15c4e"}
Oct 30 23:46:08.809801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.809747    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws" podStartSLOduration=2.80973113 podStartE2EDuration="2.80973113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:08.806570431 +0000 UTC m=+1627.733208011" watchObservedRunningTime="2026-10-30 23:46:08.80973113 +0000 UTC m=+1627.736368715"
Oct 30 23:46:08.845662 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:46:08.845638    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24982555_2ef8_4285_8db9_d9b19f8ef63e.slice/crio-a5e2955fe42b4fd93db521f801281426168c0f55cf7cc070b39db1e912135e2e WatchSource:0}: Error finding container a5e2955fe42b4fd93db521f801281426168c0f55cf7cc070b39db1e912135e2e: Status 404 returned error can't find the container with id a5e2955fe42b4fd93db521f801281426168c0f55cf7cc070b39db1e912135e2e
Oct 30 23:46:08.846279 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:08.846208    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"]
Oct 30 23:46:09.804243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:09.804202    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl" event={"ID":"24982555-2ef8-4285-8db9-d9b19f8ef63e","Type":"ContainerStarted","Data":"81b9605695f620c26d3a2f00ab163c60609999e26e5c32c147aafc76412fce31"}
Oct 30 23:46:09.804728 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:09.804274    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl" event={"ID":"24982555-2ef8-4285-8db9-d9b19f8ef63e","Type":"ContainerStarted","Data":"a5e2955fe42b4fd93db521f801281426168c0f55cf7cc070b39db1e912135e2e"}
Oct 30 23:46:09.806223 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:09.806196    4555 generic.go:356] "Generic (PLEG): container finished" podID="acec78f6-135c-4d2e-80e2-f2b70d70eb5c" containerID="3323d1d0afc2d16db881bd3f4766a294d909dc27d9b6119cb0c7f5f84ff28487" exitCode=0
Oct 30 23:46:09.806349 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:09.806239    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws" event={"ID":"acec78f6-135c-4d2e-80e2-f2b70d70eb5c","Type":"ContainerDied","Data":"3323d1d0afc2d16db881bd3f4766a294d909dc27d9b6119cb0c7f5f84ff28487"}
Oct 30 23:46:09.824346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:09.824296    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl" podStartSLOduration=2.824282224 podStartE2EDuration="2.824282224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:09.821872121 +0000 UTC m=+1628.748509726" watchObservedRunningTime="2026-10-30 23:46:09.824282224 +0000 UTC m=+1628.750919807"
Oct 30 23:46:10.821098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:10.821062    4555 generic.go:356] "Generic (PLEG): container finished" podID="24982555-2ef8-4285-8db9-d9b19f8ef63e" containerID="81b9605695f620c26d3a2f00ab163c60609999e26e5c32c147aafc76412fce31" exitCode=0
Oct 30 23:46:10.821472 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:10.821115    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl" event={"ID":"24982555-2ef8-4285-8db9-d9b19f8ef63e","Type":"ContainerDied","Data":"81b9605695f620c26d3a2f00ab163c60609999e26e5c32c147aafc76412fce31"}
Oct 30 23:46:11.050731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.050703    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:11.102078 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.101447    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-xks-ca\") pod \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") "
Oct 30 23:46:11.102078 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.101553    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-openshift-ca\") pod \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") "
Oct 30 23:46:11.102078 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.101752    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-kube-api-access-jdnfp\" (UniqueName: \"kubernetes.io/projected/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-kube-api-access-jdnfp\") pod \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\" (UID: \"acec78f6-135c-4d2e-80e2-f2b70d70eb5c\") "
Oct 30 23:46:11.106069 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.106040    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-openshift-ca" pod "acec78f6-135c-4d2e-80e2-f2b70d70eb5c" (UID: "acec78f6-135c-4d2e-80e2-f2b70d70eb5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:46:11.110875 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.110841    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-xks-ca" pod "acec78f6-135c-4d2e-80e2-f2b70d70eb5c" (UID: "acec78f6-135c-4d2e-80e2-f2b70d70eb5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:46:11.126778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.126747    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-kube-api-access-jdnfp" pod "acec78f6-135c-4d2e-80e2-f2b70d70eb5c" (UID: "acec78f6-135c-4d2e-80e2-f2b70d70eb5c"). InnerVolumeSpecName "kube-api-access-jdnfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:11.207317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.207205    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:11.207317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.207235    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:11.207317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.207246    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdnfp\" (UniqueName: \"kubernetes.io/projected/acec78f6-135c-4d2e-80e2-f2b70d70eb5c-kube-api-access-jdnfp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:11.837260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.837220    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"
Oct 30 23:46:11.837861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.837835    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws" event={"ID":"acec78f6-135c-4d2e-80e2-f2b70d70eb5c","Type":"ContainerDied","Data":"52105475ad90a8e390688d6ae6ef53f4264ffd2b8e6a5049200c49e251f15c4e"}
Oct 30 23:46:11.837955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:11.837884    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52105475ad90a8e390688d6ae6ef53f4264ffd2b8e6a5049200c49e251f15c4e"
Oct 30 23:46:12.039249 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.039208    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"]
Oct 30 23:46:12.047065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.047039    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-423201fe-cn7hs"]
Oct 30 23:46:12.108987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.108912    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:12.258395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.229989    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24982555-2ef8-4285-8db9-d9b19f8ef63e-kube-api-access-9wqd6\" (UniqueName: \"kubernetes.io/projected/24982555-2ef8-4285-8db9-d9b19f8ef63e-kube-api-access-9wqd6\") pod \"24982555-2ef8-4285-8db9-d9b19f8ef63e\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") "
Oct 30 23:46:12.258395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.230075    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-xks-ca\") pod \"24982555-2ef8-4285-8db9-d9b19f8ef63e\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") "
Oct 30 23:46:12.258395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.230219    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-openshift-ca\") pod \"24982555-2ef8-4285-8db9-d9b19f8ef63e\" (UID: \"24982555-2ef8-4285-8db9-d9b19f8ef63e\") "
Oct 30 23:46:12.258395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.238598    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-openshift-ca" pod "24982555-2ef8-4285-8db9-d9b19f8ef63e" (UID: "24982555-2ef8-4285-8db9-d9b19f8ef63e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:46:12.258395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.242433    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-xks-ca" pod "24982555-2ef8-4285-8db9-d9b19f8ef63e" (UID: "24982555-2ef8-4285-8db9-d9b19f8ef63e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:46:12.258824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.258582    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24982555-2ef8-4285-8db9-d9b19f8ef63e-kube-api-access-9wqd6" pod "24982555-2ef8-4285-8db9-d9b19f8ef63e" (UID: "24982555-2ef8-4285-8db9-d9b19f8ef63e"). InnerVolumeSpecName "kube-api-access-9wqd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:12.332791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.332761    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wqd6\" (UniqueName: \"kubernetes.io/projected/24982555-2ef8-4285-8db9-d9b19f8ef63e-kube-api-access-9wqd6\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:12.332791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.332790    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:12.332791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.332799    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24982555-2ef8-4285-8db9-d9b19f8ef63e-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:12.611004 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.610970    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"]
Oct 30 23:46:12.612005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.611985    4555 cpu_manager.go:414] "containerMap: removing container" podUID="acec78f6-135c-4d2e-80e2-f2b70d70eb5c" containerName="revoke-keys"
Oct 30 23:46:12.612126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.612007    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acec78f6-135c-4d2e-80e2-f2b70d70eb5c" containerName="revoke-keys"
Oct 30 23:46:12.612126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.612023    4555 cpu_manager.go:414] "containerMap: removing container" podUID="24982555-2ef8-4285-8db9-d9b19f8ef63e" containerName="revoke-keys"
Oct 30 23:46:12.612126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.612033    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24982555-2ef8-4285-8db9-d9b19f8ef63e" containerName="revoke-keys"
Oct 30 23:46:12.612286 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.612197    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="acec78f6-135c-4d2e-80e2-f2b70d70eb5c" containerName="revoke-keys"
Oct 30 23:46:12.612286 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.612208    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="24982555-2ef8-4285-8db9-d9b19f8ef63e" containerName="revoke-keys"
Oct 30 23:46:12.616851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.616828    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.633800 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.633449    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"]
Oct 30 23:46:12.738433 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.738393    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4a201e-mm2sj\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.738596 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.738502    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl74b\" (UniqueName: \"kubernetes.io/projected/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-kube-api-access-tl74b\") pod \"maas-api-revoke-sub-7e4a201e-mm2sj\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.738794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.738769    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4a201e-mm2sj\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.840646 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.840614    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4a201e-mm2sj\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.841132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.840750    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4a201e-mm2sj\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.841132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.840837    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl74b\" (UniqueName: \"kubernetes.io/projected/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-kube-api-access-tl74b\") pod \"maas-api-revoke-sub-7e4a201e-mm2sj\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.841684 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.841662    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4a201e-mm2sj\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.843850 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.843818    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4a201e-mm2sj\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.850489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.850466    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"
Oct 30 23:46:12.850489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.850471    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl" event={"ID":"24982555-2ef8-4285-8db9-d9b19f8ef63e","Type":"ContainerDied","Data":"a5e2955fe42b4fd93db521f801281426168c0f55cf7cc070b39db1e912135e2e"}
Oct 30 23:46:12.850656 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.850523    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e2955fe42b4fd93db521f801281426168c0f55cf7cc070b39db1e912135e2e"
Oct 30 23:46:12.855528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.855501    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl74b\" (UniqueName: \"kubernetes.io/projected/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-kube-api-access-tl74b\") pod \"maas-api-revoke-sub-7e4a201e-mm2sj\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.942443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.936029    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"]
Oct 30 23:46:12.971398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.963651    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:12.971398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.965965    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:12.971398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.967167    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"]
Oct 30 23:46:12.982159 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.982124    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6a162557\"/\"default-dockercfg-hs7gc\""
Oct 30 23:46:12.982523 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.982502    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6a162557\"/\"e2e-lifecycle-model-6a162557-kserve-self-signed-certs\""
Oct 30 23:46:12.982894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.982878    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6a162557\"/\"openshift-service-ca.crt\""
Oct 30 23:46:12.992268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:12.990750    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6a162557\"/\"kube-root-ca.crt\""
Oct 30 23:46:13.066502 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.066281    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6b4n\" (UniqueName: \"kubernetes.io/projected/761237ce-49dc-4f27-9a7e-cd5359d77257-kube-api-access-v6b4n\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.066502 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.066443    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-home\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.066765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.066548    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761237ce-49dc-4f27-9a7e-cd5359d77257-tls-certs\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.066765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.066608    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-model-cache\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.066765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.066629    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-dshm\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.066765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.066677    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-tmp-dir\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.167925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.167889    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6b4n\" (UniqueName: \"kubernetes.io/projected/761237ce-49dc-4f27-9a7e-cd5359d77257-kube-api-access-v6b4n\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.168121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.168008    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-home\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.168121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.168114    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761237ce-49dc-4f27-9a7e-cd5359d77257-tls-certs\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.168252 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.168204    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-model-cache\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.168252 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.168230    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-dshm\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.168363 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.168302    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-tmp-dir\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.168879 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.168849    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-tmp-dir\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.169076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.169053    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-model-cache\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.169559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.169531    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-home\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.171875 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.171855    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-dshm\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.197018 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.196893    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761237ce-49dc-4f27-9a7e-cd5359d77257-tls-certs\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.198819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.198791    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6b4n\" (UniqueName: \"kubernetes.io/projected/761237ce-49dc-4f27-9a7e-cd5359d77257-kube-api-access-v6b4n\") pod \"e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") " pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.283753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.229482    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b1517b-3351-4bc4-bd8f-d8dd19e32e90" path="/var/lib/kubelet/pods/d8b1517b-3351-4bc4-bd8f-d8dd19e32e90/volumes"
Oct 30 23:46:13.319209 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.319171    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:13.529122 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.529090    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"]
Oct 30 23:46:13.830588 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.830560    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"]
Oct 30 23:46:13.860852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.860821    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj" event={"ID":"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba","Type":"ContainerStarted","Data":"b253bec9733efedbfe6be9326d129bb5828eb9e231d3b1fefe47bacadcd7e6be"}
Oct 30 23:46:13.862812 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:13.862763    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd" event={"ID":"761237ce-49dc-4f27-9a7e-cd5359d77257","Type":"ContainerStarted","Data":"95d4eb4b16bdcf32246de8471092ca6f0294ad094d2fe0604a8ca4be15b97326"}
Oct 30 23:46:14.875594 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:14.875552    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd" event={"ID":"761237ce-49dc-4f27-9a7e-cd5359d77257","Type":"ContainerStarted","Data":"29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15"}
Oct 30 23:46:14.877964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:14.877934    4555 generic.go:356] "Generic (PLEG): container finished" podID="a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba" containerID="904e111112002dbfad3860ba5cad388d15bac08217ff1c6cf4cce096d99911ed" exitCode=0
Oct 30 23:46:14.878119 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:14.877975    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj" event={"ID":"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba","Type":"ContainerDied","Data":"904e111112002dbfad3860ba5cad388d15bac08217ff1c6cf4cce096d99911ed"}
Oct 30 23:46:14.896882 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:14.896824    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd" podStartSLOduration=2.896804334 podStartE2EDuration="2.896804334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:14.894664194 +0000 UTC m=+1633.821301778" watchObservedRunningTime="2026-10-30 23:46:14.896804334 +0000 UTC m=+1633.823441916"
Oct 30 23:46:15.321511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:15.321339    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:16.134249 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.134218    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:16.178463 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.173427    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-xks-ca\") pod \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") "
Oct 30 23:46:16.178463 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.173679    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-kube-api-access-tl74b\" (UniqueName: \"kubernetes.io/projected/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-kube-api-access-tl74b\") pod \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") "
Oct 30 23:46:16.178463 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.173936    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-openshift-ca\") pod \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\" (UID: \"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba\") "
Oct 30 23:46:16.178463 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.174459    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-openshift-ca" pod "a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba" (UID: "a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:46:16.178463 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.174753    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:16.179078 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.178982    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-xks-ca" pod "a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba" (UID: "a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:46:16.179078 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.179034    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-kube-api-access-tl74b" pod "a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba" (UID: "a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba"). InnerVolumeSpecName "kube-api-access-tl74b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:16.253775 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.251534    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"]
Oct 30 23:46:16.253775 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.252895    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba" containerName="revoke-keys"
Oct 30 23:46:16.253775 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.252914    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba" containerName="revoke-keys"
Oct 30 23:46:16.253775 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.253238    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba" containerName="revoke-keys"
Oct 30 23:46:16.265445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.258035    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.271026 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.270911    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bb25e9-openshift-default-dockercfg-rxp8r\""
Oct 30 23:46:16.283637 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.283600    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-socket\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.283789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.283671    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6n2c\" (UniqueName: \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-kube-api-access-m6n2c\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.283789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.283712    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25df07d8-c1dc-4594-9860-563d3ba6421a-istiod-ca-cert\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.283903 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.283887    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-data\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.283954 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.283916    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-token\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.283999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.283977    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-credential-socket\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.284043 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.284027    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-podinfo\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.284103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.284084    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-certs\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.284203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.284188    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-envoy\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.284308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.284293    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:16.284382 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.284317    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl74b\" (UniqueName: \"kubernetes.io/projected/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba-kube-api-access-tl74b\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.338495    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"]
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.385418    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6n2c\" (UniqueName: \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-kube-api-access-m6n2c\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.385483    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25df07d8-c1dc-4594-9860-563d3ba6421a-istiod-ca-cert\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.385620    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-data\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.385652    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-token\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.385710    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-credential-socket\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.385745    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-podinfo\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.385853    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-certs\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.385938    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-envoy\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.386056    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-socket\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.386563    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-socket\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.387591    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25df07d8-c1dc-4594-9860-563d3ba6421a-istiod-ca-cert\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.391507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.387842    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-data\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.427803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.420921    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-certs\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.490438 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.490332    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-credential-socket\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.561458 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.529003    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6n2c\" (UniqueName: \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-kube-api-access-m6n2c\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.643215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.643151    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-podinfo\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.665466 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.659958    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-token\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.707713 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.707682    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-envoy\") pod \"e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") " pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:16.909226 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.909195    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"
Oct 30 23:46:16.909429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.909239    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj" event={"ID":"a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba","Type":"ContainerDied","Data":"b253bec9733efedbfe6be9326d129bb5828eb9e231d3b1fefe47bacadcd7e6be"}
Oct 30 23:46:16.909429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.909289    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b253bec9733efedbfe6be9326d129bb5828eb9e231d3b1fefe47bacadcd7e6be"
Oct 30 23:46:16.928710 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:16.928677    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:17.457816 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:46:17.457784    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25df07d8_c1dc_4594_9860_563d3ba6421a.slice/crio-15f581724c44cb701df98dca322ab0593ecc5949cb4947815185c4ff0636ec3c WatchSource:0}: Error finding container 15f581724c44cb701df98dca322ab0593ecc5949cb4947815185c4ff0636ec3c: Status 404 returned error can't find the container with id 15f581724c44cb701df98dca322ab0593ecc5949cb4947815185c4ff0636ec3c
Oct 30 23:46:17.459265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.459244    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"]
Oct 30 23:46:17.461230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.461196    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:46:17.461331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.461262    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:46:17.461331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.461292    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:46:17.920396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.920332    4555 generic.go:356] "Generic (PLEG): container finished" podID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerID="3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3" exitCode=0
Oct 30 23:46:17.920576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.920413    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" event={"ID":"c2029c09-8186-46dd-bc4f-aab88bde8fd1","Type":"ContainerDied","Data":"3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3"}
Oct 30 23:46:17.920947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.920925    4555 scope.go:122] "RemoveContainer" containerID="3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3"
Oct 30 23:46:17.922714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.922690    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" event={"ID":"25df07d8-c1dc-4594-9860-563d3ba6421a","Type":"ContainerStarted","Data":"7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739"}
Oct 30 23:46:17.922834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.922739    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" event={"ID":"25df07d8-c1dc-4594-9860-563d3ba6421a","Type":"ContainerStarted","Data":"15f581724c44cb701df98dca322ab0593ecc5949cb4947815185c4ff0636ec3c"}
Oct 30 23:46:17.922908 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.922884    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:46:17.922967 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.922954    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:46:17.923025 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.922993    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:46:17.930023 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:17.929545    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:18.030424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:18.001956    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" podStartSLOduration=2.001937781 podStartE2EDuration="2.001937781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:17.996746908 +0000 UTC m=+1636.923384506" watchObservedRunningTime="2026-10-30 23:46:18.001937781 +0000 UTC m=+1636.928575364"
Oct 30 23:46:18.933141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:18.933103    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" event={"ID":"c2029c09-8186-46dd-bc4f-aab88bde8fd1","Type":"ContainerStarted","Data":"84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78"}
Oct 30 23:46:18.937106 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:18.937080    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:19.183812 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:19.183724    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:46:19.185126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:19.185097    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 23:46:19.185278 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:19.185150    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 23:46:19.942140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:19.942108    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:19.942881 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:19.942863    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:46:20.192444 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:20.191751    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 23:46:20.192444 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:20.191814    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 23:46:20.783567 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:20.783471    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:46:21.185868 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:21.185743    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 23:46:21.186466 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:21.186432    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 23:46:22.208886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:22.208850    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 23:46:22.209533 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:22.209502    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 23:46:23.184819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:23.184785    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 23:46:23.185267 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:23.185220    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 23:46:23.324515 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:23.323425    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:23.338721 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:23.338697    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:24.184768 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:24.184730    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 23:46:24.185040 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:24.185012    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 23:46:24.309594 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:24.309563    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:46:25.183850 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:25.183814    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 23:46:25.184505 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:25.184462    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 23:46:26.188767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:26.188729    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:46:26.190060 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:26.190036    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 30 23:46:27.447669 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:46:27.447633    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:46:33.329671 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:33.328454    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:46:33.505527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:33.505476    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:46:33.638080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:33.637948    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 23:46:33.773512 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:33.773460    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:46:42.043594 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:42.043514    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"]
Oct 30 23:46:42.048632 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:42.048588    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b442ad33-sk8wb"]
Oct 30 23:46:43.209973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:43.209940    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62c9467-b445-44fb-b292-e554bfb80f51" path="/var/lib/kubelet/pods/e62c9467-b445-44fb-b292-e554bfb80f51/volumes"
Oct 30 23:46:46.029183 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:46.029149    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"]
Oct 30 23:46:46.033538 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:46.033511    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ada8fa5-kcklg"]
Oct 30 23:46:47.225396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:47.221970    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93eae285-b8d0-4f49-b72a-d6de913834ee" path="/var/lib/kubelet/pods/93eae285-b8d0-4f49-b72a-d6de913834ee/volumes"
Oct 30 23:46:52.146028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.145997    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"]
Oct 30 23:46:52.150635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.150614    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:46:52.156191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.156166    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b6cb71\""
Oct 30 23:46:52.162504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.162480    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"]
Oct 30 23:46:52.305164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.305124    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e306d720-bdba-4c26-9c4d-6f12a36d6989-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg\" (UID: \"e306d720-bdba-4c26-9c4d-6f12a36d6989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:46:52.404538 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.404453    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2rkm\" (UniqueName: \"kubernetes.io/projected/e306d720-bdba-4c26-9c4d-6f12a36d6989-kube-api-access-m2rkm\") pod \"payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg\" (UID: \"e306d720-bdba-4c26-9c4d-6f12a36d6989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:46:52.514397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.510363    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e306d720-bdba-4c26-9c4d-6f12a36d6989-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg\" (UID: \"e306d720-bdba-4c26-9c4d-6f12a36d6989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:46:52.514397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.510453    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2rkm\" (UniqueName: \"kubernetes.io/projected/e306d720-bdba-4c26-9c4d-6f12a36d6989-kube-api-access-m2rkm\") pod \"payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg\" (UID: \"e306d720-bdba-4c26-9c4d-6f12a36d6989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:46:52.514397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.511821    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e306d720-bdba-4c26-9c4d-6f12a36d6989-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg\" (UID: \"e306d720-bdba-4c26-9c4d-6f12a36d6989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:46:52.557680 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.557645    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2rkm\" (UniqueName: \"kubernetes.io/projected/e306d720-bdba-4c26-9c4d-6f12a36d6989-kube-api-access-m2rkm\") pod \"payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg\" (UID: \"e306d720-bdba-4c26-9c4d-6f12a36d6989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:46:52.780531 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:52.780504    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:46:53.124509 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:53.124469    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"]
Oct 30 23:46:53.125193 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:46:53.125163    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode306d720_bdba_4c26_9c4d_6f12a36d6989.slice/crio-3bf4e968b3fd3d8754e994db533312871ebcd022f4c6746e3ac458ecefa1b9b0 WatchSource:0}: Error finding container 3bf4e968b3fd3d8754e994db533312871ebcd022f4c6746e3ac458ecefa1b9b0: Status 404 returned error can't find the container with id 3bf4e968b3fd3d8754e994db533312871ebcd022f4c6746e3ac458ecefa1b9b0
Oct 30 23:46:53.724164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:53.724128    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg" event={"ID":"e306d720-bdba-4c26-9c4d-6f12a36d6989","Type":"ContainerStarted","Data":"99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636"}
Oct 30 23:46:53.724615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:53.724187    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg" event={"ID":"e306d720-bdba-4c26-9c4d-6f12a36d6989","Type":"ContainerStarted","Data":"3bf4e968b3fd3d8754e994db533312871ebcd022f4c6746e3ac458ecefa1b9b0"}
Oct 30 23:46:53.724615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:53.724228    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:46:53.747286 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:46:53.747239    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg" podStartSLOduration=1.747225313 podStartE2EDuration="1.747225313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:53.742782152 +0000 UTC m=+1672.669419770" watchObservedRunningTime="2026-10-30 23:46:53.747225313 +0000 UTC m=+1672.673862894"
Oct 30 23:46:57.516392 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:46:57.516014    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:47:00.060395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:00.056426    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"]
Oct 30 23:47:00.086394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:00.081607    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba7a0243-z44vz"]
Oct 30 23:47:01.242667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:01.242627    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dfa9cfa-d1f1-4589-80c7-71409236eef8" path="/var/lib/kubelet/pods/9dfa9cfa-d1f1-4589-80c7-71409236eef8/volumes"
Oct 30 23:47:03.318817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:03.318787    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:47:04.758950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:04.758902    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:47:07.647398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:07.598144    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 23:47:07.681007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:07.638465    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:07.764404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:07.764333    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:07.862408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:07.846469    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Oct 30 23:47:08.012799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.012695    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:08.087530 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.087479    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:08.132584 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.132523    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:47:08.270589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.270550    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd" podUID="761237ce-49dc-4f27-9a7e-cd5359d77257" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:08.366395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.344887    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:08.368095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.368060    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:47:08.368359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.368332    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:08.710619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.709475    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:08.711144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.710797    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 23:47:08.715411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.711949    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:47:08.715411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.712013    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:47:08.777995 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.776804    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:47:08.783393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:08.778545    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:47:09.041754 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.041641    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:09.042305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.042279    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:47:09.069420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.068529    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:47:09.069420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.068602    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:09.107393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.098241    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 23:47:09.107393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.102531    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:47:09.152480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.133441    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:09.152480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.133877    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:47:09.732053 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.732023    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:47:09.741482 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.741456    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:47:09.742853 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.741708    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:47:09.743070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.743053    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 30 23:47:09.743193 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.739137    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:47:09.804711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:09.799789    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:47:10.411403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:10.408439    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:47:17.998712 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:17.998665    4555 generic.go:356] "Generic (PLEG): container finished" podID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerID="7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739" exitCode=0
Oct 30 23:47:17.999320 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:17.999009    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" event={"ID":"25df07d8-c1dc-4594-9860-563d3ba6421a","Type":"ContainerDied","Data":"7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739"}
Oct 30 23:47:17.999652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:17.999621    4555 scope.go:122] "RemoveContainer" containerID="7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739"
Oct 30 23:47:18.004707 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:18.004639    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:47:18.011111 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:18.011070    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:47:18.011400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:18.011358    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:47:19.011181 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:19.011141    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" event={"ID":"25df07d8-c1dc-4594-9860-563d3ba6421a","Type":"ContainerStarted","Data":"3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228"}
Oct 30 23:47:19.919121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:19.919079    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"]
Oct 30 23:47:19.922390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:19.922350    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:19.928708 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:19.926615    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:47:19.928708 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:19.927737    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:47:19.931095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:19.930434    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:47:19.931775 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:19.931543    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"]
Oct 30 23:47:19.932466 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:19.932443    4555 patch_prober.go:29] interesting pod/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:47:19.932572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:19.932486    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:47:19.941761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:19.941734    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:47:20.040951 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.040915    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:20.041409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.040960    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:20.041409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.041168    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgcs\" (UniqueName: \"kubernetes.io/projected/7b8d834e-8e05-4836-8aae-a76b6d30d600-kube-api-access-6cgcs\") pod \"maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:20.142910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.142877    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgcs\" (UniqueName: \"kubernetes.io/projected/7b8d834e-8e05-4836-8aae-a76b6d30d600-kube-api-access-6cgcs\") pod \"maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:20.143082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.143052    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:20.143129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.143085    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:20.143783 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.143752    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:20.145917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.145894    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:20.155413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.155389    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgcs\" (UniqueName: \"kubernetes.io/projected/7b8d834e-8e05-4836-8aae-a76b6d30d600-kube-api-access-6cgcs\") pod \"maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:20.239527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.239437    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:20.611159 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:47:20.611099    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8d834e_8e05_4836_8aae_a76b6d30d600.slice/crio-c18bd54d0ecc99d25c8710e481dd7d59794992150552b305a9220409bffeb0e4 WatchSource:0}: Error finding container c18bd54d0ecc99d25c8710e481dd7d59794992150552b305a9220409bffeb0e4: Status 404 returned error can't find the container with id c18bd54d0ecc99d25c8710e481dd7d59794992150552b305a9220409bffeb0e4
Oct 30 23:47:20.613955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.613905    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"]
Oct 30 23:47:20.930563 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.930527    4555 patch_prober.go:29] interesting pod/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:47:20.930706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:20.930581    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:47:21.034797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:21.034755    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s" event={"ID":"7b8d834e-8e05-4836-8aae-a76b6d30d600","Type":"ContainerStarted","Data":"c18bd54d0ecc99d25c8710e481dd7d59794992150552b305a9220409bffeb0e4"}
Oct 30 23:47:21.930010 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:21.929976    4555 patch_prober.go:29] interesting pod/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:47:21.930428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:21.930061    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:47:22.055729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:22.055692    4555 generic.go:356] "Generic (PLEG): container finished" podID="7b8d834e-8e05-4836-8aae-a76b6d30d600" containerID="7f4c076c12d5f30f0f8273a7f9582ef01f3cb925a8805b959a8e9ebb0293e97f" exitCode=0
Oct 30 23:47:22.055902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:22.055878    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s" event={"ID":"7b8d834e-8e05-4836-8aae-a76b6d30d600","Type":"ContainerDied","Data":"7f4c076c12d5f30f0f8273a7f9582ef01f3cb925a8805b959a8e9ebb0293e97f"}
Oct 30 23:47:22.929852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:22.929818    4555 patch_prober.go:29] interesting pod/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:47:22.930037 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:22.929872    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:47:23.324062 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.323672    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:23.491397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.480076    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8d834e-8e05-4836-8aae-a76b6d30d600-kube-api-access-6cgcs\" (UniqueName: \"kubernetes.io/projected/7b8d834e-8e05-4836-8aae-a76b6d30d600-kube-api-access-6cgcs\") pod \"7b8d834e-8e05-4836-8aae-a76b6d30d600\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") "
Oct 30 23:47:23.491397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.480407    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-xks-ca\") pod \"7b8d834e-8e05-4836-8aae-a76b6d30d600\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") "
Oct 30 23:47:23.491397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.480539    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-openshift-ca\") pod \"7b8d834e-8e05-4836-8aae-a76b6d30d600\" (UID: \"7b8d834e-8e05-4836-8aae-a76b6d30d600\") "
Oct 30 23:47:23.532024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.529103    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8d834e-8e05-4836-8aae-a76b6d30d600-kube-api-access-6cgcs" pod "7b8d834e-8e05-4836-8aae-a76b6d30d600" (UID: "7b8d834e-8e05-4836-8aae-a76b6d30d600"). InnerVolumeSpecName "kube-api-access-6cgcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:47:23.532211 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.532176    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-openshift-ca" pod "7b8d834e-8e05-4836-8aae-a76b6d30d600" (UID: "7b8d834e-8e05-4836-8aae-a76b6d30d600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:47:23.533638 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.533606    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-xks-ca" pod "7b8d834e-8e05-4836-8aae-a76b6d30d600" (UID: "7b8d834e-8e05-4836-8aae-a76b6d30d600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:23.591408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.591096    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:23.591408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.591138    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8d834e-8e05-4836-8aae-a76b6d30d600-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:23.591408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.591157    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cgcs\" (UniqueName: \"kubernetes.io/projected/7b8d834e-8e05-4836-8aae-a76b6d30d600-kube-api-access-6cgcs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:23.929603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.929518    4555 patch_prober.go:29] interesting pod/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:47:23.929603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:23.929588    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:47:24.025780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:24.025743    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"]
Oct 30 23:47:24.030518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:24.030487    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b62f526-n6hhs"]
Oct 30 23:47:24.143767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:24.142951    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s" event={"ID":"7b8d834e-8e05-4836-8aae-a76b6d30d600","Type":"ContainerDied","Data":"c18bd54d0ecc99d25c8710e481dd7d59794992150552b305a9220409bffeb0e4"}
Oct 30 23:47:24.143767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:24.143058    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c18bd54d0ecc99d25c8710e481dd7d59794992150552b305a9220409bffeb0e4"
Oct 30 23:47:24.143767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:24.143136    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"
Oct 30 23:47:24.935065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:24.935033    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:47:24.936243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:24.936217    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:47:25.212022 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:25.211941    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1859d0e-496f-49d0-ad41-aa1a84cc424f" path="/var/lib/kubelet/pods/e1859d0e-496f-49d0-ad41-aa1a84cc424f/volumes"
Oct 30 23:47:27.453239 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:47:27.453194    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:47:29.030340 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:29.030288    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"]
Oct 30 23:47:29.039930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:29.039889    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44fb6a5-xt494"]
Oct 30 23:47:29.208726 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:29.208690    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b501a4-4b1f-4e82-8000-9ea14bb48386" path="/var/lib/kubelet/pods/72b501a4-4b1f-4e82-8000-9ea14bb48386/volumes"
Oct 30 23:47:34.024549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:34.024514    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"]
Oct 30 23:47:34.029348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:34.029318    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a806a1a-z9ngh"]
Oct 30 23:47:35.212416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:35.212356    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a16667-9aa5-4686-9c72-98b099cb6282" path="/var/lib/kubelet/pods/d0a16667-9aa5-4686-9c72-98b099cb6282/volumes"
Oct 30 23:47:47.950236 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:47.950195    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"]
Oct 30 23:47:47.955798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:47.955764    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bb25e9-4vz5s"]
Oct 30 23:47:48.027052 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.027013    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"]
Oct 30 23:47:48.031490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.031469    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5af5fce-w4vnj"]
Oct 30 23:47:48.243697 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.243619    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"]
Oct 30 23:47:48.244310 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.244296    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7b8d834e-8e05-4836-8aae-a76b6d30d600" containerName="revoke-keys"
Oct 30 23:47:48.244357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.244311    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8d834e-8e05-4836-8aae-a76b6d30d600" containerName="revoke-keys"
Oct 30 23:47:48.244508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.244497    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8d834e-8e05-4836-8aae-a76b6d30d600" containerName="revoke-keys"
Oct 30 23:47:48.250166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.250102    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.254101 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.254075    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:47:48.255289 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.255265    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:47:48.256528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.256502    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"]
Oct 30 23:47:48.406798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.406764    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d67dc42-lwttt\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.406978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.406914    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d67dc42-lwttt\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.407043 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.407022    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7bc\" (UniqueName: \"kubernetes.io/projected/7072f6a6-53fe-45cc-bae1-35637abda998-kube-api-access-nw7bc\") pod \"maas-api-revoke-sub-2d67dc42-lwttt\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.508615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.508514    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7bc\" (UniqueName: \"kubernetes.io/projected/7072f6a6-53fe-45cc-bae1-35637abda998-kube-api-access-nw7bc\") pod \"maas-api-revoke-sub-2d67dc42-lwttt\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.508796 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.508780    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d67dc42-lwttt\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.508931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.508906    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d67dc42-lwttt\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.509494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.509472    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d67dc42-lwttt\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.511464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.511431    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d67dc42-lwttt\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.527700 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.527670    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7bc\" (UniqueName: \"kubernetes.io/projected/7072f6a6-53fe-45cc-bae1-35637abda998-kube-api-access-nw7bc\") pod \"maas-api-revoke-sub-2d67dc42-lwttt\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.580275 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.580232    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:48.871256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:48.871221    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"]
Oct 30 23:47:49.212499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:49.212409    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8d834e-8e05-4836-8aae-a76b6d30d600" path="/var/lib/kubelet/pods/7b8d834e-8e05-4836-8aae-a76b6d30d600/volumes"
Oct 30 23:47:49.213038 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:49.213020    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba3b5bf-02e9-43a9-830e-6aecc3162d0c" path="/var/lib/kubelet/pods/eba3b5bf-02e9-43a9-830e-6aecc3162d0c/volumes"
Oct 30 23:47:49.456876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:49.456831    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt" event={"ID":"7072f6a6-53fe-45cc-bae1-35637abda998","Type":"ContainerStarted","Data":"ba1f1d763d65c65c47f8fa5321ba9e85f40158b37c8e984bd43e2a2b87611252"}
Oct 30 23:47:49.457080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:49.456896    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt" event={"ID":"7072f6a6-53fe-45cc-bae1-35637abda998","Type":"ContainerStarted","Data":"83cf6298c1d2ff485676e445bc71ef4535b9be45a3eae1e167ab3a0bf9fd5c74"}
Oct 30 23:47:49.480602 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:49.480485    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt" podStartSLOduration=1.480465958 podStartE2EDuration="1.480465958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:49.474871952 +0000 UTC m=+1728.401509534" watchObservedRunningTime="2026-10-30 23:47:49.480465958 +0000 UTC m=+1728.407103540"
Oct 30 23:47:50.470119 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:50.470089    4555 generic.go:356] "Generic (PLEG): container finished" podID="7072f6a6-53fe-45cc-bae1-35637abda998" containerID="ba1f1d763d65c65c47f8fa5321ba9e85f40158b37c8e984bd43e2a2b87611252" exitCode=0
Oct 30 23:47:50.470644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:50.470172    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt" event={"ID":"7072f6a6-53fe-45cc-bae1-35637abda998","Type":"ContainerDied","Data":"ba1f1d763d65c65c47f8fa5321ba9e85f40158b37c8e984bd43e2a2b87611252"}
Oct 30 23:47:51.648758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:51.648723    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:51.769341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:51.769251    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-xks-ca\") pod \"7072f6a6-53fe-45cc-bae1-35637abda998\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") "
Oct 30 23:47:51.769535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:51.769502    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7072f6a6-53fe-45cc-bae1-35637abda998-kube-api-access-nw7bc\" (UniqueName: \"kubernetes.io/projected/7072f6a6-53fe-45cc-bae1-35637abda998-kube-api-access-nw7bc\") pod \"7072f6a6-53fe-45cc-bae1-35637abda998\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") "
Oct 30 23:47:51.769535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:51.769530    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-openshift-ca\") pod \"7072f6a6-53fe-45cc-bae1-35637abda998\" (UID: \"7072f6a6-53fe-45cc-bae1-35637abda998\") "
Oct 30 23:47:51.770019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:51.769948    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-openshift-ca" pod "7072f6a6-53fe-45cc-bae1-35637abda998" (UID: "7072f6a6-53fe-45cc-bae1-35637abda998"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:47:51.770417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:51.770398    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:51.771606 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:51.771583    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-xks-ca" pod "7072f6a6-53fe-45cc-bae1-35637abda998" (UID: "7072f6a6-53fe-45cc-bae1-35637abda998"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:51.771851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:51.771837    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7072f6a6-53fe-45cc-bae1-35637abda998-kube-api-access-nw7bc" pod "7072f6a6-53fe-45cc-bae1-35637abda998" (UID: "7072f6a6-53fe-45cc-bae1-35637abda998"). InnerVolumeSpecName "kube-api-access-nw7bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:47:51.871690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:51.871660    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7bc\" (UniqueName: \"kubernetes.io/projected/7072f6a6-53fe-45cc-bae1-35637abda998-kube-api-access-nw7bc\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:51.871690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:51.871688    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7072f6a6-53fe-45cc-bae1-35637abda998-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:52.488138 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:52.488107    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"
Oct 30 23:47:52.488290 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:52.488139    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt" event={"ID":"7072f6a6-53fe-45cc-bae1-35637abda998","Type":"ContainerDied","Data":"83cf6298c1d2ff485676e445bc71ef4535b9be45a3eae1e167ab3a0bf9fd5c74"}
Oct 30 23:47:52.488290 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:52.488191    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83cf6298c1d2ff485676e445bc71ef4535b9be45a3eae1e167ab3a0bf9fd5c74"
Oct 30 23:47:54.120397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:54.120030    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"]
Oct 30 23:47:54.158322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:54.158265    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07e3845f-r2z9k"]
Oct 30 23:47:55.212009 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:55.211972    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414683df-56f9-4d37-af39-3eee805e1000" path="/var/lib/kubelet/pods/414683df-56f9-4d37-af39-3eee805e1000/volumes"
Oct 30 23:47:55.938449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:55.937429    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11079-7e86ce"]
Oct 30 23:47:55.946525 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:55.946494    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7072f6a6-53fe-45cc-bae1-35637abda998" containerName="revoke-keys"
Oct 30 23:47:55.946764 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:55.946747    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7072f6a6-53fe-45cc-bae1-35637abda998" containerName="revoke-keys"
Oct 30 23:47:55.972704 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:55.947152    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="7072f6a6-53fe-45cc-bae1-35637abda998" containerName="revoke-keys"
Oct 30 23:47:55.987656 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:55.986620    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-7e86ce"
Oct 30 23:47:55.998633 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:55.998607    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-csqxw\""
Oct 30 23:47:56.067913 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:56.067874    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11079-7e86ce"]
Oct 30 23:47:56.082212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:56.081784    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4fz\" (UniqueName: \"kubernetes.io/projected/63c055ab-55aa-4d32-acda-3c05fb084ba4-kube-api-access-wd4fz\") pod \"test-curl-11079-7e86ce\" (UID: \"63c055ab-55aa-4d32-acda-3c05fb084ba4\") " pod="opendatahub/test-curl-11079-7e86ce"
Oct 30 23:47:56.183784 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:56.183735    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4fz\" (UniqueName: \"kubernetes.io/projected/63c055ab-55aa-4d32-acda-3c05fb084ba4-kube-api-access-wd4fz\") pod \"test-curl-11079-7e86ce\" (UID: \"63c055ab-55aa-4d32-acda-3c05fb084ba4\") " pod="opendatahub/test-curl-11079-7e86ce"
Oct 30 23:47:56.197230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:56.197142    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4fz\" (UniqueName: \"kubernetes.io/projected/63c055ab-55aa-4d32-acda-3c05fb084ba4-kube-api-access-wd4fz\") pod \"test-curl-11079-7e86ce\" (UID: \"63c055ab-55aa-4d32-acda-3c05fb084ba4\") " pod="opendatahub/test-curl-11079-7e86ce"
Oct 30 23:47:56.368307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:56.368274    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-7e86ce"
Oct 30 23:47:56.664083 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:47:56.664050    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63c055ab_55aa_4d32_acda_3c05fb084ba4.slice/crio-9c044d1ba33e11a3c45f6f16d319daa4ff9ba7f07e44bae8793b81de60eea291 WatchSource:0}: Error finding container 9c044d1ba33e11a3c45f6f16d319daa4ff9ba7f07e44bae8793b81de60eea291: Status 404 returned error can't find the container with id 9c044d1ba33e11a3c45f6f16d319daa4ff9ba7f07e44bae8793b81de60eea291
Oct 30 23:47:56.664315 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:56.664297    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11079-7e86ce"]
Oct 30 23:47:56.666746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:56.666727    4555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:47:57.337403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:57.336164    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"]
Oct 30 23:47:57.337403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:57.336450    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy" containerID="cri-o://3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228" gracePeriod=30
Oct 30 23:47:57.630454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:57.629944    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11079-7e86ce" event={"ID":"63c055ab-55aa-4d32-acda-3c05fb084ba4","Type":"ContainerStarted","Data":"9c044d1ba33e11a3c45f6f16d319daa4ff9ba7f07e44bae8793b81de60eea291"}
Oct 30 23:47:57.726393 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:47:57.719582    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:47:57.913666 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:57.913291    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"]
Oct 30 23:47:57.919082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:57.917850    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:57.928408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:57.928361    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"]
Oct 30 23:47:57.929553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:57.929523    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:47:57.935395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:57.935358    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:47:58.054026 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.053993    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:58.054263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.054237    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shxx\" (UniqueName: \"kubernetes.io/projected/366066ca-a3ff-4013-a7f0-c03daa16d502-kube-api-access-8shxx\") pod \"maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:58.054405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.054364    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:58.157413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.156557    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:58.157413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.156838    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shxx\" (UniqueName: \"kubernetes.io/projected/366066ca-a3ff-4013-a7f0-c03daa16d502-kube-api-access-8shxx\") pod \"maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:58.157413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.156927    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:58.157413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.157306    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:58.159932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.159911    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:58.176318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.176259    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shxx\" (UniqueName: \"kubernetes.io/projected/366066ca-a3ff-4013-a7f0-c03daa16d502-kube-api-access-8shxx\") pod \"maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:58.245868 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.245837    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:47:58.580456 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:47:58.574559    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366066ca_a3ff_4013_a7f0_c03daa16d502.slice/crio-195b877289e375b2bf9f5e675b75a8dd9423e5f9fe1efd6a1a25e2cc28249b29 WatchSource:0}: Error finding container 195b877289e375b2bf9f5e675b75a8dd9423e5f9fe1efd6a1a25e2cc28249b29: Status 404 returned error can't find the container with id 195b877289e375b2bf9f5e675b75a8dd9423e5f9fe1efd6a1a25e2cc28249b29
Oct 30 23:47:58.580456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.577140    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"]
Oct 30 23:47:58.642629 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:58.642594    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz" event={"ID":"366066ca-a3ff-4013-a7f0-c03daa16d502","Type":"ContainerStarted","Data":"195b877289e375b2bf9f5e675b75a8dd9423e5f9fe1efd6a1a25e2cc28249b29"}
Oct 30 23:47:59.730937 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:47:59.730662    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz" event={"ID":"366066ca-a3ff-4013-a7f0-c03daa16d502","Type":"ContainerStarted","Data":"e964fb2ba9299f6bce3f1c91b6838377ddf6c1f2eb4209dd4667661c159f8596"}
Oct 30 23:48:00.916527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:00.911689    4555 generic.go:356] "Generic (PLEG): container finished" podID="366066ca-a3ff-4013-a7f0-c03daa16d502" containerID="e964fb2ba9299f6bce3f1c91b6838377ddf6c1f2eb4209dd4667661c159f8596" exitCode=0
Oct 30 23:48:00.916527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:00.911992    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz" event={"ID":"366066ca-a3ff-4013-a7f0-c03daa16d502","Type":"ContainerDied","Data":"e964fb2ba9299f6bce3f1c91b6838377ddf6c1f2eb4209dd4667661c159f8596"}
Oct 30 23:48:01.922442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:01.922405    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11079-7e86ce" event={"ID":"63c055ab-55aa-4d32-acda-3c05fb084ba4","Type":"ContainerStarted","Data":"d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb"}
Oct 30 23:48:01.950396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:01.947825    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11079-7e86ce" podStartSLOduration=2.402384419 podStartE2EDuration="6.947805127s" totalImagesPullingTime="4.545420708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:01.944356705 +0000 UTC m=+1740.870994287" watchObservedRunningTime="2026-10-30 23:48:01.947805127 +0000 UTC m=+1740.874442712"
Oct 30 23:48:02.105504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.105477    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:48:02.260453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.256095    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366066ca-a3ff-4013-a7f0-c03daa16d502-kube-api-access-8shxx\" (UniqueName: \"kubernetes.io/projected/366066ca-a3ff-4013-a7f0-c03daa16d502-kube-api-access-8shxx\") pod \"366066ca-a3ff-4013-a7f0-c03daa16d502\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") "
Oct 30 23:48:02.260453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.256318    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-openshift-ca\") pod \"366066ca-a3ff-4013-a7f0-c03daa16d502\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") "
Oct 30 23:48:02.307614 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.307570    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-openshift-ca" pod "366066ca-a3ff-4013-a7f0-c03daa16d502" (UID: "366066ca-a3ff-4013-a7f0-c03daa16d502"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:02.308492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.308458    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-xks-ca\") pod \"366066ca-a3ff-4013-a7f0-c03daa16d502\" (UID: \"366066ca-a3ff-4013-a7f0-c03daa16d502\") "
Oct 30 23:48:02.327395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.326631    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366066ca-a3ff-4013-a7f0-c03daa16d502-kube-api-access-8shxx" pod "366066ca-a3ff-4013-a7f0-c03daa16d502" (UID: "366066ca-a3ff-4013-a7f0-c03daa16d502"). InnerVolumeSpecName "kube-api-access-8shxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:02.328362 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.328304    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.328524 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.328364    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8shxx\" (UniqueName: \"kubernetes.io/projected/366066ca-a3ff-4013-a7f0-c03daa16d502-kube-api-access-8shxx\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.347496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.347414    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-xks-ca" pod "366066ca-a3ff-4013-a7f0-c03daa16d502" (UID: "366066ca-a3ff-4013-a7f0-c03daa16d502"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:48:02.431774 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.431269    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366066ca-a3ff-4013-a7f0-c03daa16d502-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.515448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.513262    4555 scope.go:122] "RemoveContainer" containerID="5f686030639209c05204e03a6b1dd024a2d86083b72ede640b8b6cf4200e56d5"
Oct 30 23:48:02.563250 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.563226    4555 scope.go:122] "RemoveContainer" containerID="a654c43afec7524c57608022129406482a7c95fafebc5d8e6430f46dbae65436"
Oct 30 23:48:02.581581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.579928    4555 scope.go:122] "RemoveContainer" containerID="9bbe3738328d4227a080ee038a5706c2df6cb8defc6461f5d93ba8bbb4012506"
Oct 30 23:48:02.596579 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.596478    4555 scope.go:122] "RemoveContainer" containerID="7c2eb48cc6f234c3c0af3bfdd7ebe9b78f55241ce9eff95ee78318450684997a"
Oct 30 23:48:02.774989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.774500    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:48:02.857961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.853109    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-kube-api-access-m6n2c\" (UniqueName: \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-kube-api-access-m6n2c\") pod \"25df07d8-c1dc-4594-9860-563d3ba6421a\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") "
Oct 30 23:48:02.857961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.853163    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25df07d8-c1dc-4594-9860-563d3ba6421a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25df07d8-c1dc-4594-9860-563d3ba6421a-istiod-ca-cert\") pod \"25df07d8-c1dc-4594-9860-563d3ba6421a\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") "
Oct 30 23:48:02.857961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.853224    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-podinfo\") pod \"25df07d8-c1dc-4594-9860-563d3ba6421a\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") "
Oct 30 23:48:02.857961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.853311    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-envoy\") pod \"25df07d8-c1dc-4594-9860-563d3ba6421a\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") "
Oct 30 23:48:02.857961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.853348    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-socket\") pod \"25df07d8-c1dc-4594-9860-563d3ba6421a\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") "
Oct 30 23:48:02.857961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.853435    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-token\" (UniqueName: \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-token\") pod \"25df07d8-c1dc-4594-9860-563d3ba6421a\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") "
Oct 30 23:48:02.857961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.853488    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-certs\") pod \"25df07d8-c1dc-4594-9860-563d3ba6421a\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") "
Oct 30 23:48:02.857961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.853700    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-credential-socket\") pod \"25df07d8-c1dc-4594-9860-563d3ba6421a\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") "
Oct 30 23:48:02.857961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.853732    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-data\") pod \"25df07d8-c1dc-4594-9860-563d3ba6421a\" (UID: \"25df07d8-c1dc-4594-9860-563d3ba6421a\") "
Oct 30 23:48:02.859279 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.858982    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25df07d8-c1dc-4594-9860-563d3ba6421a-istiod-ca-cert" pod "25df07d8-c1dc-4594-9860-563d3ba6421a" (UID: "25df07d8-c1dc-4594-9860-563d3ba6421a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:02.867416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.860474    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-envoy" pod "25df07d8-c1dc-4594-9860-563d3ba6421a" (UID: "25df07d8-c1dc-4594-9860-563d3ba6421a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:48:02.872109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.868786    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-data" pod "25df07d8-c1dc-4594-9860-563d3ba6421a" (UID: "25df07d8-c1dc-4594-9860-563d3ba6421a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:48:02.873220 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.872807    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.873220 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.872838    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.873220 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.872872    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25df07d8-c1dc-4594-9860-563d3ba6421a-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.885392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.873606    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-socket" pod "25df07d8-c1dc-4594-9860-563d3ba6421a" (UID: "25df07d8-c1dc-4594-9860-563d3ba6421a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:48:02.886159 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.885779    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-credential-socket" pod "25df07d8-c1dc-4594-9860-563d3ba6421a" (UID: "25df07d8-c1dc-4594-9860-563d3ba6421a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:48:02.892895 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.886390    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-certs" pod "25df07d8-c1dc-4594-9860-563d3ba6421a" (UID: "25df07d8-c1dc-4594-9860-563d3ba6421a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:48:02.901698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.901659    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-kube-api-access-m6n2c" pod "25df07d8-c1dc-4594-9860-563d3ba6421a" (UID: "25df07d8-c1dc-4594-9860-563d3ba6421a"). InnerVolumeSpecName "kube-api-access-m6n2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:02.903091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.903048    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-podinfo" pod "25df07d8-c1dc-4594-9860-563d3ba6421a" (UID: "25df07d8-c1dc-4594-9860-563d3ba6421a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:48:02.903091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.903067    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-token" pod "25df07d8-c1dc-4594-9860-563d3ba6421a" (UID: "25df07d8-c1dc-4594-9860-563d3ba6421a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:02.935916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.935887    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"
Oct 30 23:48:02.936405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.935876    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz" event={"ID":"366066ca-a3ff-4013-a7f0-c03daa16d502","Type":"ContainerDied","Data":"195b877289e375b2bf9f5e675b75a8dd9423e5f9fe1efd6a1a25e2cc28249b29"}
Oct 30 23:48:02.936405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.936055    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195b877289e375b2bf9f5e675b75a8dd9423e5f9fe1efd6a1a25e2cc28249b29"
Oct 30 23:48:02.939184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.938208    4555 generic.go:356] "Generic (PLEG): container finished" podID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerID="3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228" exitCode=0
Oct 30 23:48:02.939184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.938284    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"
Oct 30 23:48:02.939184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.938299    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" event={"ID":"25df07d8-c1dc-4594-9860-563d3ba6421a","Type":"ContainerDied","Data":"3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228"}
Oct 30 23:48:02.939184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.938363    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw" event={"ID":"25df07d8-c1dc-4594-9860-563d3ba6421a","Type":"ContainerDied","Data":"15f581724c44cb701df98dca322ab0593ecc5949cb4947815185c4ff0636ec3c"}
Oct 30 23:48:02.939184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.938421    4555 scope.go:122] "RemoveContainer" containerID="3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228"
Oct 30 23:48:02.954442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.954419    4555 scope.go:122] "RemoveContainer" containerID="7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739"
Oct 30 23:48:02.968446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.967800    4555 scope.go:122] "RemoveContainer" containerID="3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228"
Oct 30 23:48:02.968446 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:48:02.968153    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228\": container with ID starting with 3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228 not found: ID does not exist" containerID="3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228"
Oct 30 23:48:02.968446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.968186    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228"} err="failed to get container status \"3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228\": rpc error: code = NotFound desc = could not find container \"3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228\": container with ID starting with 3c11ca5149224e9abe6e7c954fd516aea80b4cc31e8e9de698801192b0f66228 not found: ID does not exist"
Oct 30 23:48:02.968446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.968208    4555 scope.go:122] "RemoveContainer" containerID="7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739"
Oct 30 23:48:02.968677 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:48:02.968500    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739\": container with ID starting with 7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739 not found: ID does not exist" containerID="7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739"
Oct 30 23:48:02.968677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.968524    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739"} err="failed to get container status \"7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739\": rpc error: code = NotFound desc = could not find container \"7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739\": container with ID starting with 7a9182367b1fdcf8c36d94947ef01755d539f3f0417ad7ec56c68470bc745739 not found: ID does not exist"
Oct 30 23:48:02.973886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.973865    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"]
Oct 30 23:48:02.975031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.975006    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.975141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.975033    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.975141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.975049    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.975141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.975062    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25df07d8-c1dc-4594-9860-563d3ba6421a-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.975141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.975075    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6n2c\" (UniqueName: \"kubernetes.io/projected/25df07d8-c1dc-4594-9860-563d3ba6421a-kube-api-access-m6n2c\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.975141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.975090    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25df07d8-c1dc-4594-9860-563d3ba6421a-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:02.985978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:02.985957    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bb25e9-openshift-default-86984885c9-sdjbw"]
Oct 30 23:48:03.064625 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.064529    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11079-7e86ce"]
Oct 30 23:48:03.073324 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.073289    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11079-7e86ce"]
Oct 30 23:48:03.211872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.211839    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" path="/var/lib/kubelet/pods/25df07d8-c1dc-4594-9860-563d3ba6421a/volumes"
Oct 30 23:48:03.217603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.217575    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"]
Oct 30 23:48:03.218220 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218206    4555 cpu_manager.go:414] "containerMap: removing container" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy"
Oct 30 23:48:03.218280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218223    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy"
Oct 30 23:48:03.218280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218251    4555 cpu_manager.go:414] "containerMap: removing container" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy"
Oct 30 23:48:03.218280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218257    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy"
Oct 30 23:48:03.218280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218266    4555 cpu_manager.go:414] "containerMap: removing container" podUID="63c055ab-55aa-4d32-acda-3c05fb084ba4" containerName="test-curl-11079-7e86ce"
Oct 30 23:48:03.218280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218271    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c055ab-55aa-4d32-acda-3c05fb084ba4" containerName="test-curl-11079-7e86ce"
Oct 30 23:48:03.218462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218287    4555 cpu_manager.go:414] "containerMap: removing container" podUID="366066ca-a3ff-4013-a7f0-c03daa16d502" containerName="revoke-keys"
Oct 30 23:48:03.218462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218292    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366066ca-a3ff-4013-a7f0-c03daa16d502" containerName="revoke-keys"
Oct 30 23:48:03.218462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218406    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c055ab-55aa-4d32-acda-3c05fb084ba4" containerName="test-curl-11079-7e86ce"
Oct 30 23:48:03.218462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218419    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="366066ca-a3ff-4013-a7f0-c03daa16d502" containerName="revoke-keys"
Oct 30 23:48:03.218462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218430    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy"
Oct 30 23:48:03.218462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.218438    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="25df07d8-c1dc-4594-9860-563d3ba6421a" containerName="istio-proxy"
Oct 30 23:48:03.223869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.223851    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 30 23:48:03.228926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.228895    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-4e6c3b\""
Oct 30 23:48:03.229121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.229100    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-4e6c3b-dockercfg-8pzpm\""
Oct 30 23:48:03.233442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.233417    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"]
Oct 30 23:48:03.279682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.279646    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"]
Oct 30 23:48:03.280552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.280524    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9\" (UID: \"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 30 23:48:03.281586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.280763    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpwv\" (UniqueName: \"kubernetes.io/projected/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-kube-api-access-4fpwv\") pod \"payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9\" (UID: \"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 30 23:48:03.284799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.284780    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 30 23:48:03.295785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.295762    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"]
Oct 30 23:48:03.412441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.408639    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9\" (UID: \"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 30 23:48:03.412441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.408763    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpwv\" (UniqueName: \"kubernetes.io/projected/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-kube-api-access-4fpwv\") pod \"payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9\" (UID: \"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 30 23:48:03.412441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.410215    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9\" (UID: \"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 30 23:48:03.438234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.435692    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpwv\" (UniqueName: \"kubernetes.io/projected/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-kube-api-access-4fpwv\") pod \"payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9\" (UID: \"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 30 23:48:03.524239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.523346    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2918e4a9-cdfb-460f-82f5-941944ce7a47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4\" (UID: \"2918e4a9-cdfb-460f-82f5-941944ce7a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 30 23:48:03.524239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.523469    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dpl4\" (UniqueName: \"kubernetes.io/projected/2918e4a9-cdfb-460f-82f5-941944ce7a47-kube-api-access-6dpl4\") pod \"payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4\" (UID: \"2918e4a9-cdfb-460f-82f5-941944ce7a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 30 23:48:03.544907 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.544648    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 30 23:48:03.627463 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.626280    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2918e4a9-cdfb-460f-82f5-941944ce7a47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4\" (UID: \"2918e4a9-cdfb-460f-82f5-941944ce7a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 30 23:48:03.627674 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.627650    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2918e4a9-cdfb-460f-82f5-941944ce7a47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4\" (UID: \"2918e4a9-cdfb-460f-82f5-941944ce7a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 30 23:48:03.627952 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.627830    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dpl4\" (UniqueName: \"kubernetes.io/projected/2918e4a9-cdfb-460f-82f5-941944ce7a47-kube-api-access-6dpl4\") pod \"payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4\" (UID: \"2918e4a9-cdfb-460f-82f5-941944ce7a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 30 23:48:03.641797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.641732    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dpl4\" (UniqueName: \"kubernetes.io/projected/2918e4a9-cdfb-460f-82f5-941944ce7a47-kube-api-access-6dpl4\") pod \"payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4\" (UID: \"2918e4a9-cdfb-460f-82f5-941944ce7a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 30 23:48:03.843357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.843325    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"]
Oct 30 23:48:03.850577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.850544    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:03.852189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.851954    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"]
Oct 30 23:48:03.854655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.854626    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:48:03.854746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.854701    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:48:03.904611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.904581    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 30 23:48:03.938533 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.938502    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-dxklv\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:03.939186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.938834    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsjsx\" (UniqueName: \"kubernetes.io/projected/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-kube-api-access-bsjsx\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-dxklv\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:03.939186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.938938    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-dxklv\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:03.944939 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:48:03.944912    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cd0b30_12b8_4c3a_b1ca_dba8525e7190.slice/crio-9c7a07b364dd8daf238041160f34a537a7d8d1a823fae4fa1430048bc05e9426 WatchSource:0}: Error finding container 9c7a07b364dd8daf238041160f34a537a7d8d1a823fae4fa1430048bc05e9426: Status 404 returned error can't find the container with id 9c7a07b364dd8daf238041160f34a537a7d8d1a823fae4fa1430048bc05e9426
Oct 30 23:48:03.947599 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.947527    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"]
Oct 30 23:48:03.951520 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:03.951493    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11079-7e86ce" podUID="63c055ab-55aa-4d32-acda-3c05fb084ba4" containerName="test-curl-11079-7e86ce" containerID="cri-o://d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb" gracePeriod=2
Oct 30 23:48:04.078289 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.078154    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsjsx\" (UniqueName: \"kubernetes.io/projected/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-kube-api-access-bsjsx\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-dxklv\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:04.080583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.079731    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-dxklv\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:04.080583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.079888    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-dxklv\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:04.082161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.082097    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-dxklv\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:04.086841 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.086776    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-dxklv\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:04.110607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.110534    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"]
Oct 30 23:48:04.114479 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.114347    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b52579-dpp2n"]
Oct 30 23:48:04.120605 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.120333    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsjsx\" (UniqueName: \"kubernetes.io/projected/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-kube-api-access-bsjsx\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-dxklv\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:04.174404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.174341    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:04.206418 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:48:04.206385    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2918e4a9_cdfb_460f_82f5_941944ce7a47.slice/crio-acafffbc37c533243bcfe1d36a848d39917f122e8df8b5e84797ae46d54e3ed9 WatchSource:0}: Error finding container acafffbc37c533243bcfe1d36a848d39917f122e8df8b5e84797ae46d54e3ed9: Status 404 returned error can't find the container with id acafffbc37c533243bcfe1d36a848d39917f122e8df8b5e84797ae46d54e3ed9
Oct 30 23:48:04.207872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.207846    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"]
Oct 30 23:48:04.466490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.466461    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"]
Oct 30 23:48:04.466738 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:48:04.466718    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412cbf1d_cb7a_452f_9435_41f3b7b5bec4.slice/crio-53b7a569ce57dfce3054fa7b80a1fe93a8c0bd8ae2950ff572eb0a5e2401f012 WatchSource:0}: Error finding container 53b7a569ce57dfce3054fa7b80a1fe93a8c0bd8ae2950ff572eb0a5e2401f012: Status 404 returned error can't find the container with id 53b7a569ce57dfce3054fa7b80a1fe93a8c0bd8ae2950ff572eb0a5e2401f012
Oct 30 23:48:04.962652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.962615    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4" event={"ID":"2918e4a9-cdfb-460f-82f5-941944ce7a47","Type":"ContainerStarted","Data":"682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74"}
Oct 30 23:48:04.963133 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.962712    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4" event={"ID":"2918e4a9-cdfb-460f-82f5-941944ce7a47","Type":"ContainerStarted","Data":"acafffbc37c533243bcfe1d36a848d39917f122e8df8b5e84797ae46d54e3ed9"}
Oct 30 23:48:04.963497 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.963472    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 30 23:48:04.965593 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.965565    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv" event={"ID":"412cbf1d-cb7a-452f-9435-41f3b7b5bec4","Type":"ContainerStarted","Data":"f4d4312b7e0c50e45e904ae079aeb55ad9e9cb78d5f12664e44c9a93b9c74fbd"}
Oct 30 23:48:04.965733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.965611    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv" event={"ID":"412cbf1d-cb7a-452f-9435-41f3b7b5bec4","Type":"ContainerStarted","Data":"53b7a569ce57dfce3054fa7b80a1fe93a8c0bd8ae2950ff572eb0a5e2401f012"}
Oct 30 23:48:04.967345 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.967322    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9" event={"ID":"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190","Type":"ContainerStarted","Data":"28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa"}
Oct 30 23:48:04.967485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.967383    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9" event={"ID":"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190","Type":"ContainerStarted","Data":"9c7a07b364dd8daf238041160f34a537a7d8d1a823fae4fa1430048bc05e9426"}
Oct 30 23:48:04.967485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.967413    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 30 23:48:04.987962 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:04.987913    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4" podStartSLOduration=1.987896309 podStartE2EDuration="1.987896309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:04.986024019 +0000 UTC m=+1743.912661603" watchObservedRunningTime="2026-10-30 23:48:04.987896309 +0000 UTC m=+1743.914533890"
Oct 30 23:48:05.013397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:05.013328    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv" podStartSLOduration=2.013313945 podStartE2EDuration="2.013313945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:05.00921676 +0000 UTC m=+1743.935854344" watchObservedRunningTime="2026-10-30 23:48:05.013313945 +0000 UTC m=+1743.939951526"
Oct 30 23:48:05.031596 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:05.031507    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9" podStartSLOduration=2.031494288 podStartE2EDuration="2.031494288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:05.029610784 +0000 UTC m=+1743.956248365" watchObservedRunningTime="2026-10-30 23:48:05.031494288 +0000 UTC m=+1743.958131868"
Oct 30 23:48:05.319607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:05.312739    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2ee303-1b62-47cc-bc03-c1dce4cf2647" path="/var/lib/kubelet/pods/cf2ee303-1b62-47cc-bc03-c1dce4cf2647/volumes"
Oct 30 23:48:05.982885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:05.982856    4555 generic.go:356] "Generic (PLEG): container finished" podID="412cbf1d-cb7a-452f-9435-41f3b7b5bec4" containerID="f4d4312b7e0c50e45e904ae079aeb55ad9e9cb78d5f12664e44c9a93b9c74fbd" exitCode=0
Oct 30 23:48:05.983283 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:05.982944    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv" event={"ID":"412cbf1d-cb7a-452f-9435-41f3b7b5bec4","Type":"ContainerDied","Data":"f4d4312b7e0c50e45e904ae079aeb55ad9e9cb78d5f12664e44c9a93b9c74fbd"}
Oct 30 23:48:06.178952 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.178925    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-7e86ce"
Oct 30 23:48:06.270356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.270321    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c055ab-55aa-4d32-acda-3c05fb084ba4-kube-api-access-wd4fz\" (UniqueName: \"kubernetes.io/projected/63c055ab-55aa-4d32-acda-3c05fb084ba4-kube-api-access-wd4fz\") pod \"63c055ab-55aa-4d32-acda-3c05fb084ba4\" (UID: \"63c055ab-55aa-4d32-acda-3c05fb084ba4\") "
Oct 30 23:48:06.277909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.277875    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c055ab-55aa-4d32-acda-3c05fb084ba4-kube-api-access-wd4fz" pod "63c055ab-55aa-4d32-acda-3c05fb084ba4" (UID: "63c055ab-55aa-4d32-acda-3c05fb084ba4"). InnerVolumeSpecName "kube-api-access-wd4fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:06.373595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.373560    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd4fz\" (UniqueName: \"kubernetes.io/projected/63c055ab-55aa-4d32-acda-3c05fb084ba4-kube-api-access-wd4fz\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:06.531527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.531445    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"]
Oct 30 23:48:06.536129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.536111    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:48:06.543671 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.543647    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2ccd42\""
Oct 30 23:48:06.543852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.543647    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2ccd42-dockercfg-8xfst\""
Oct 30 23:48:06.550436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.550408    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"]
Oct 30 23:48:06.589783 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.589751    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"]
Oct 30 23:48:06.596539 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.596511    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:48:06.599158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.599134    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"]
Oct 30 23:48:06.770489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.770453    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbb7\" (UniqueName: \"kubernetes.io/projected/85fac7c9-babe-422f-96f4-d58352606a54-kube-api-access-wsbb7\") pod \"payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv\" (UID: \"85fac7c9-babe-422f-96f4-d58352606a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:48:06.864306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.859300    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fac7c9-babe-422f-96f4-d58352606a54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv\" (UID: \"85fac7c9-babe-422f-96f4-d58352606a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:48:06.882391 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.882049    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gpv\" (UniqueName: \"kubernetes.io/projected/e76a92f7-ee45-4e29-bac6-2d47c6557755-kube-api-access-j4gpv\") pod \"payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm\" (UID: \"e76a92f7-ee45-4e29-bac6-2d47c6557755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:48:06.889617 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.886739    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76a92f7-ee45-4e29-bac6-2d47c6557755-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm\" (UID: \"e76a92f7-ee45-4e29-bac6-2d47c6557755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:48:06.997815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.995329    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gpv\" (UniqueName: \"kubernetes.io/projected/e76a92f7-ee45-4e29-bac6-2d47c6557755-kube-api-access-j4gpv\") pod \"payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm\" (UID: \"e76a92f7-ee45-4e29-bac6-2d47c6557755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:48:06.997815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.995581    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76a92f7-ee45-4e29-bac6-2d47c6557755-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm\" (UID: \"e76a92f7-ee45-4e29-bac6-2d47c6557755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:48:06.997815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.995690    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbb7\" (UniqueName: \"kubernetes.io/projected/85fac7c9-babe-422f-96f4-d58352606a54-kube-api-access-wsbb7\") pod \"payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv\" (UID: \"85fac7c9-babe-422f-96f4-d58352606a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:48:06.997815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.995823    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fac7c9-babe-422f-96f4-d58352606a54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv\" (UID: \"85fac7c9-babe-422f-96f4-d58352606a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:48:06.997815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.996646    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fac7c9-babe-422f-96f4-d58352606a54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv\" (UID: \"85fac7c9-babe-422f-96f4-d58352606a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:48:06.997815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.997288    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76a92f7-ee45-4e29-bac6-2d47c6557755-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm\" (UID: \"e76a92f7-ee45-4e29-bac6-2d47c6557755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:48:06.999498 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.999469    4555 generic.go:356] "Generic (PLEG): container finished" podID="63c055ab-55aa-4d32-acda-3c05fb084ba4" containerID="d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb" exitCode=137
Oct 30 23:48:06.999631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.999510    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-7e86ce"
Oct 30 23:48:06.999631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:06.999563    4555 scope.go:122] "RemoveContainer" containerID="d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb"
Oct 30 23:48:07.010173 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.010145    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gpv\" (UniqueName: \"kubernetes.io/projected/e76a92f7-ee45-4e29-bac6-2d47c6557755-kube-api-access-j4gpv\") pod \"payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm\" (UID: \"e76a92f7-ee45-4e29-bac6-2d47c6557755\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:48:07.012038 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.010997    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbb7\" (UniqueName: \"kubernetes.io/projected/85fac7c9-babe-422f-96f4-d58352606a54-kube-api-access-wsbb7\") pod \"payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv\" (UID: \"85fac7c9-babe-422f-96f4-d58352606a54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:48:07.014625 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.014528    4555 scope.go:122] "RemoveContainer" containerID="d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb"
Oct 30 23:48:07.014897 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:48:07.014877    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb\": container with ID starting with d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb not found: ID does not exist" containerID="d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb"
Oct 30 23:48:07.014966 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.014909    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb"} err="failed to get container status \"d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb\": rpc error: code = NotFound desc = could not find container \"d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb\": container with ID starting with d56d0e08ebac12afef0d75f1835c9ffbcf98704fe6a467dff2542188ac83cefb not found: ID does not exist"
Oct 30 23:48:07.039711 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:48:07.039672    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11079-7e86ce\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="63c055ab-55aa-4d32-acda-3c05fb084ba4" pod="opendatahub/test-curl-11079-7e86ce"
Oct 30 23:48:07.152680 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.152397    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:48:07.155024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.154998    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:48:07.216188 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.215704    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c055ab-55aa-4d32-acda-3c05fb084ba4" path="/var/lib/kubelet/pods/63c055ab-55aa-4d32-acda-3c05fb084ba4/volumes"
Oct 30 23:48:07.228246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.228225    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:07.302440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.302407    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-openshift-ca\") pod \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") "
Oct 30 23:48:07.302609 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.302460    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-kube-api-access-bsjsx\" (UniqueName: \"kubernetes.io/projected/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-kube-api-access-bsjsx\") pod \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") "
Oct 30 23:48:07.302680 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.302642    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-xks-ca\") pod \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\" (UID: \"412cbf1d-cb7a-452f-9435-41f3b7b5bec4\") "
Oct 30 23:48:07.303983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.303954    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-openshift-ca" pod "412cbf1d-cb7a-452f-9435-41f3b7b5bec4" (UID: "412cbf1d-cb7a-452f-9435-41f3b7b5bec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:07.311003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.310964    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-kube-api-access-bsjsx" pod "412cbf1d-cb7a-452f-9435-41f3b7b5bec4" (UID: "412cbf1d-cb7a-452f-9435-41f3b7b5bec4"). InnerVolumeSpecName "kube-api-access-bsjsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:07.311535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.311509    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-xks-ca" pod "412cbf1d-cb7a-452f-9435-41f3b7b5bec4" (UID: "412cbf1d-cb7a-452f-9435-41f3b7b5bec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:48:07.405733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.405640    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:07.405733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.405680    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:07.405733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.405698    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsjsx\" (UniqueName: \"kubernetes.io/projected/412cbf1d-cb7a-452f-9435-41f3b7b5bec4-kube-api-access-bsjsx\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:07.515659 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:48:07.515623    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76a92f7_ee45_4e29_bac6_2d47c6557755.slice/crio-f0541a00ddf1920fb256056c670e9018d646de036eb996e690bc4d8c205d5f90 WatchSource:0}: Error finding container f0541a00ddf1920fb256056c670e9018d646de036eb996e690bc4d8c205d5f90: Status 404 returned error can't find the container with id f0541a00ddf1920fb256056c670e9018d646de036eb996e690bc4d8c205d5f90
Oct 30 23:48:07.515785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.515657    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"]
Oct 30 23:48:07.521696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:07.521670    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"]
Oct 30 23:48:07.521823 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:48:07.521786    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fac7c9_babe_422f_96f4_d58352606a54.slice/crio-ac20f694bb7b0692f16bc6c76ea806253390bf4dfc9f01c2a1904636710a0856 WatchSource:0}: Error finding container ac20f694bb7b0692f16bc6c76ea806253390bf4dfc9f01c2a1904636710a0856: Status 404 returned error can't find the container with id ac20f694bb7b0692f16bc6c76ea806253390bf4dfc9f01c2a1904636710a0856
Oct 30 23:48:08.013640 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.013600    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv" event={"ID":"85fac7c9-babe-422f-96f4-d58352606a54","Type":"ContainerStarted","Data":"f1b39fa07183fc2e80e89e844e2654efd42a331e269085e05a95726dded168cb"}
Oct 30 23:48:08.014186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.013667    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv" event={"ID":"85fac7c9-babe-422f-96f4-d58352606a54","Type":"ContainerStarted","Data":"ac20f694bb7b0692f16bc6c76ea806253390bf4dfc9f01c2a1904636710a0856"}
Oct 30 23:48:08.014186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.013720    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:48:08.016017 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.015991    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm" event={"ID":"e76a92f7-ee45-4e29-bac6-2d47c6557755","Type":"ContainerStarted","Data":"0ba0902feca7af97586366411a63507cf3a77e8de8c872d7a592119c32ddda59"}
Oct 30 23:48:08.016159 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.016143    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:48:08.016246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.016175    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm" event={"ID":"e76a92f7-ee45-4e29-bac6-2d47c6557755","Type":"ContainerStarted","Data":"f0541a00ddf1920fb256056c670e9018d646de036eb996e690bc4d8c205d5f90"}
Oct 30 23:48:08.017760 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.017736    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv" event={"ID":"412cbf1d-cb7a-452f-9435-41f3b7b5bec4","Type":"ContainerDied","Data":"53b7a569ce57dfce3054fa7b80a1fe93a8c0bd8ae2950ff572eb0a5e2401f012"}
Oct 30 23:48:08.017883 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.017785    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"
Oct 30 23:48:08.017883 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.017836    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53b7a569ce57dfce3054fa7b80a1fe93a8c0bd8ae2950ff572eb0a5e2401f012"
Oct 30 23:48:08.034519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.034483    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv" podStartSLOduration=2.034470309 podStartE2EDuration="2.034470309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:08.033187162 +0000 UTC m=+1746.959824744" watchObservedRunningTime="2026-10-30 23:48:08.034470309 +0000 UTC m=+1746.961107890"
Oct 30 23:48:08.056015 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.055987    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"]
Oct 30 23:48:08.075865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:08.075807    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm" podStartSLOduration=2.07578971 podStartE2EDuration="2.07578971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:08.069564765 +0000 UTC m=+1746.996202347" watchObservedRunningTime="2026-10-30 23:48:08.07578971 +0000 UTC m=+1747.002427295"
Oct 30 23:48:10.661140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.661105    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"]
Oct 30 23:48:10.665409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.662097    4555 cpu_manager.go:414] "containerMap: removing container" podUID="412cbf1d-cb7a-452f-9435-41f3b7b5bec4" containerName="revoke-keys"
Oct 30 23:48:10.665409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.662118    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412cbf1d-cb7a-452f-9435-41f3b7b5bec4" containerName="revoke-keys"
Oct 30 23:48:10.665409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.662312    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="412cbf1d-cb7a-452f-9435-41f3b7b5bec4" containerName="revoke-keys"
Oct 30 23:48:10.690392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.668835    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.698333 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.693330    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-200408-serving-cert\""
Oct 30 23:48:10.710028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.710006    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"]
Oct 30 23:48:10.779771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.779737    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rp7x\" (UniqueName: \"kubernetes.io/projected/a1c131d4-4156-40bf-9f1e-e62079bd51a8-kube-api-access-8rp7x\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.779973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.779948    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.780100 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.780084    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c131d4-4156-40bf-9f1e-e62079bd51a8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.780139 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.780130    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.883400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.882538    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rp7x\" (UniqueName: \"kubernetes.io/projected/a1c131d4-4156-40bf-9f1e-e62079bd51a8-kube-api-access-8rp7x\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.884151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.884121    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.884286 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.884256    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c131d4-4156-40bf-9f1e-e62079bd51a8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.884348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.884304    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.884986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.884961    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c131d4-4156-40bf-9f1e-e62079bd51a8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.892466 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.892440    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.895109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.895079    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:10.900227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:10.900204    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rp7x\" (UniqueName: \"kubernetes.io/projected/a1c131d4-4156-40bf-9f1e-e62079bd51a8-kube-api-access-8rp7x\") pod \"maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:11.025878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:11.025852    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:11.400494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:11.380326    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"]
Oct 30 23:48:12.072084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:12.072047    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl" event={"ID":"a1c131d4-4156-40bf-9f1e-e62079bd51a8","Type":"ContainerStarted","Data":"8a9226c36097c245a7953701404cffadbe67876cf7fdeb814479963559b81e93"}
Oct 30 23:48:13.057133 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.057089    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg" podUID="e306d720-bdba-4c26-9c4d-6f12a36d6989" containerName="payload-pre-processing" containerID="cri-o://99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636" gracePeriod=25
Oct 30 23:48:13.085274 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.085246    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl" event={"ID":"a1c131d4-4156-40bf-9f1e-e62079bd51a8","Type":"ContainerStarted","Data":"df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772"}
Oct 30 23:48:13.085666 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.085418    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:13.109846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.109796    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl" podStartSLOduration=2.675892531 podStartE2EDuration="3.10978346s" totalImagesPullingTime="433.890929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:13.108133256 +0000 UTC m=+1752.034770838" watchObservedRunningTime="2026-10-30 23:48:13.10978346 +0000 UTC m=+1752.036421040"
Oct 30 23:48:13.450165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.450134    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:48:13.644397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.644333    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e306d720-bdba-4c26-9c4d-6f12a36d6989-kube-api-access-m2rkm\" (UniqueName: \"kubernetes.io/projected/e306d720-bdba-4c26-9c4d-6f12a36d6989-kube-api-access-m2rkm\") pod \"e306d720-bdba-4c26-9c4d-6f12a36d6989\" (UID: \"e306d720-bdba-4c26-9c4d-6f12a36d6989\") "
Oct 30 23:48:13.644568 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.644481    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e306d720-bdba-4c26-9c4d-6f12a36d6989-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e306d720-bdba-4c26-9c4d-6f12a36d6989-plugins-config-volume\") pod \"e306d720-bdba-4c26-9c4d-6f12a36d6989\" (UID: \"e306d720-bdba-4c26-9c4d-6f12a36d6989\") "
Oct 30 23:48:13.644813 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.644783    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e306d720-bdba-4c26-9c4d-6f12a36d6989-plugins-config-volume" pod "e306d720-bdba-4c26-9c4d-6f12a36d6989" (UID: "e306d720-bdba-4c26-9c4d-6f12a36d6989"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:13.645007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.644984    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e306d720-bdba-4c26-9c4d-6f12a36d6989-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:13.646723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.646701    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e306d720-bdba-4c26-9c4d-6f12a36d6989-kube-api-access-m2rkm" pod "e306d720-bdba-4c26-9c4d-6f12a36d6989" (UID: "e306d720-bdba-4c26-9c4d-6f12a36d6989"). InnerVolumeSpecName "kube-api-access-m2rkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:13.746842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:13.746758    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2rkm\" (UniqueName: \"kubernetes.io/projected/e306d720-bdba-4c26-9c4d-6f12a36d6989-kube-api-access-m2rkm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:14.064756 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:14.064714    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"]
Oct 30 23:48:14.134424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:14.134382    4555 generic.go:356] "Generic (PLEG): container finished" podID="e306d720-bdba-4c26-9c4d-6f12a36d6989" containerID="99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636" exitCode=0
Oct 30 23:48:14.135331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:14.135313    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"
Oct 30 23:48:14.137045 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:14.136842    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg" event={"ID":"e306d720-bdba-4c26-9c4d-6f12a36d6989","Type":"ContainerDied","Data":"99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636"}
Oct 30 23:48:14.137045 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:14.136944    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg" event={"ID":"e306d720-bdba-4c26-9c4d-6f12a36d6989","Type":"ContainerDied","Data":"3bf4e968b3fd3d8754e994db533312871ebcd022f4c6746e3ac458ecefa1b9b0"}
Oct 30 23:48:14.137045 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:14.136975    4555 scope.go:122] "RemoveContainer" containerID="99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636"
Oct 30 23:48:14.152459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:14.152429    4555 scope.go:122] "RemoveContainer" containerID="99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636"
Oct 30 23:48:14.153040 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:48:14.153009    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636\": container with ID starting with 99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636 not found: ID does not exist" containerID="99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636"
Oct 30 23:48:14.153140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:14.153052    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636"} err="failed to get container status \"99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636\": rpc error: code = NotFound desc = could not find container \"99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636\": container with ID starting with 99e091bee38c9c315d986549370202156cad5c8279c8c0e0f76d04db28806636 not found: ID does not exist"
Oct 30 23:48:14.176141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:14.175508    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"]
Oct 30 23:48:14.180942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:14.180916    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b6cb71-595db56ccf-j56vg"]
Oct 30 23:48:15.027138 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:15.027101    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"]
Oct 30 23:48:15.031361 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:15.031340    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72992f85-9g58w"]
Oct 30 23:48:15.213341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:15.213305    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc83e902-4d5a-46cc-b0ac-724c397fd528" path="/var/lib/kubelet/pods/dc83e902-4d5a-46cc-b0ac-724c397fd528/volumes"
Oct 30 23:48:15.213985 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:15.213967    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e306d720-bdba-4c26-9c4d-6f12a36d6989" path="/var/lib/kubelet/pods/e306d720-bdba-4c26-9c4d-6f12a36d6989/volumes"
Oct 30 23:48:15.986698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:15.986668    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 30 23:48:16.582337 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:16.582308    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:48:17.003956 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:17.003867    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 30 23:48:19.071398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:19.054646    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:48:19.105396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:19.078587    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:48:19.105396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:19.079988    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw" podUID="be309a26-4bd4-4f00-927c-f9b0fb48ca31" containerName="payload-processing" containerID="cri-o://cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62" gracePeriod=25
Oct 30 23:48:19.218403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:19.211022    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 30 23:48:19.342451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:19.328905    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:48:20.156858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:20.156814    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw" podUID="be309a26-4bd4-4f00-927c-f9b0fb48ca31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:48:21.411433 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:21.410591    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:48:21.511393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:21.508454    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:48:26.857935 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.857901    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"]
Oct 30 23:48:26.858715 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.858697    4555 cpu_manager.go:414] "containerMap: removing container" podUID="e306d720-bdba-4c26-9c4d-6f12a36d6989" containerName="payload-pre-processing"
Oct 30 23:48:26.858715 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.858717    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e306d720-bdba-4c26-9c4d-6f12a36d6989" containerName="payload-pre-processing"
Oct 30 23:48:26.858860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.858850    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="e306d720-bdba-4c26-9c4d-6f12a36d6989" containerName="payload-pre-processing"
Oct 30 23:48:26.862636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.862612    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:26.867263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.867207    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:48:26.876326 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.876303    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:48:26.896897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.896852    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"]
Oct 30 23:48:26.989212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.989178    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6dm\" (UniqueName: \"kubernetes.io/projected/ec03e339-604b-4ef8-8646-957af49d386e-kube-api-access-mj6dm\") pod \"maas-api-revoke-sub-e131ad43-ck7sl\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:26.989450 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.989421    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e131ad43-ck7sl\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:26.989567 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:26.989549    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e131ad43-ck7sl\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:27.091994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:27.091808    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e131ad43-ck7sl\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:27.091994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:27.091916    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e131ad43-ck7sl\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:27.092236 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:27.092168    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6dm\" (UniqueName: \"kubernetes.io/projected/ec03e339-604b-4ef8-8646-957af49d386e-kube-api-access-mj6dm\") pod \"maas-api-revoke-sub-e131ad43-ck7sl\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:27.093417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:27.093389    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e131ad43-ck7sl\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:27.096605 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:27.096547    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e131ad43-ck7sl\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:27.110051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:27.109932    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6dm\" (UniqueName: \"kubernetes.io/projected/ec03e339-604b-4ef8-8646-957af49d386e-kube-api-access-mj6dm\") pod \"maas-api-revoke-sub-e131ad43-ck7sl\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:27.206767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:27.206723    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:27.446610 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:48:27.446516    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:48:27.507510 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:27.507476    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"]
Oct 30 23:48:27.508974 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:48:27.508945    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec03e339_604b_4ef8_8646_957af49d386e.slice/crio-0b9b3bd244bf4d832988b7eff2c0c8bf1ba730b32203f786daf2ba3508307a5b WatchSource:0}: Error finding container 0b9b3bd244bf4d832988b7eff2c0c8bf1ba730b32203f786daf2ba3508307a5b: Status 404 returned error can't find the container with id 0b9b3bd244bf4d832988b7eff2c0c8bf1ba730b32203f786daf2ba3508307a5b
Oct 30 23:48:27.636756 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:27.636716    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl" event={"ID":"ec03e339-604b-4ef8-8646-957af49d386e","Type":"ContainerStarted","Data":"0b9b3bd244bf4d832988b7eff2c0c8bf1ba730b32203f786daf2ba3508307a5b"}
Oct 30 23:48:28.646554 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:28.646518    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl" event={"ID":"ec03e339-604b-4ef8-8646-957af49d386e","Type":"ContainerStarted","Data":"c592548c6a72aa7e905ebd605aea3b144d76ee84df4aa46f37ac03948325c712"}
Oct 30 23:48:28.665458 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:28.665411    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl" podStartSLOduration=2.665356245 podStartE2EDuration="2.665356245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:28.663878664 +0000 UTC m=+1767.590516248" watchObservedRunningTime="2026-10-30 23:48:28.665356245 +0000 UTC m=+1767.591993825"
Oct 30 23:48:29.659594 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:29.659560    4555 generic.go:356] "Generic (PLEG): container finished" podID="ec03e339-604b-4ef8-8646-957af49d386e" containerID="c592548c6a72aa7e905ebd605aea3b144d76ee84df4aa46f37ac03948325c712" exitCode=0
Oct 30 23:48:29.660071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:29.659638    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl" event={"ID":"ec03e339-604b-4ef8-8646-957af49d386e","Type":"ContainerDied","Data":"c592548c6a72aa7e905ebd605aea3b144d76ee84df4aa46f37ac03948325c712"}
Oct 30 23:48:30.102838 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:30.102797    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw" podUID="be309a26-4bd4-4f00-927c-f9b0fb48ca31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:48:30.955404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:30.954747    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:31.039204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.039176    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec03e339-604b-4ef8-8646-957af49d386e-kube-api-access-mj6dm\" (UniqueName: \"kubernetes.io/projected/ec03e339-604b-4ef8-8646-957af49d386e-kube-api-access-mj6dm\") pod \"ec03e339-604b-4ef8-8646-957af49d386e\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") "
Oct 30 23:48:31.039445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.039222    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-openshift-ca\") pod \"ec03e339-604b-4ef8-8646-957af49d386e\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") "
Oct 30 23:48:31.039903 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.039871    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-openshift-ca" pod "ec03e339-604b-4ef8-8646-957af49d386e" (UID: "ec03e339-604b-4ef8-8646-957af49d386e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:31.042251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.042231    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec03e339-604b-4ef8-8646-957af49d386e-kube-api-access-mj6dm" pod "ec03e339-604b-4ef8-8646-957af49d386e" (UID: "ec03e339-604b-4ef8-8646-957af49d386e"). InnerVolumeSpecName "kube-api-access-mj6dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:31.141413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.141362    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-xks-ca\") pod \"ec03e339-604b-4ef8-8646-957af49d386e\" (UID: \"ec03e339-604b-4ef8-8646-957af49d386e\") "
Oct 30 23:48:31.141997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.141974    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj6dm\" (UniqueName: \"kubernetes.io/projected/ec03e339-604b-4ef8-8646-957af49d386e-kube-api-access-mj6dm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:31.141997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.141999    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:31.143574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.143538    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-xks-ca" pod "ec03e339-604b-4ef8-8646-957af49d386e" (UID: "ec03e339-604b-4ef8-8646-957af49d386e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:48:31.252027 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.251772    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec03e339-604b-4ef8-8646-957af49d386e-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:31.720394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.718816    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl" event={"ID":"ec03e339-604b-4ef8-8646-957af49d386e","Type":"ContainerDied","Data":"0b9b3bd244bf4d832988b7eff2c0c8bf1ba730b32203f786daf2ba3508307a5b"}
Oct 30 23:48:31.720394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.718877    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9b3bd244bf4d832988b7eff2c0c8bf1ba730b32203f786daf2ba3508307a5b"
Oct 30 23:48:31.720394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:31.718947    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"
Oct 30 23:48:33.223534 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.223502    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"]
Oct 30 23:48:33.224562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.224544    4555 cpu_manager.go:414] "containerMap: removing container" podUID="ec03e339-604b-4ef8-8646-957af49d386e" containerName="revoke-keys"
Oct 30 23:48:33.224638 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.224566    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec03e339-604b-4ef8-8646-957af49d386e" containerName="revoke-keys"
Oct 30 23:48:33.224775 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.224764    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec03e339-604b-4ef8-8646-957af49d386e" containerName="revoke-keys"
Oct 30 23:48:33.230151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.230128    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.245443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.241900    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-885999-openshift-default-dockercfg-m96bw\""
Oct 30 23:48:33.287382 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.287334    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"]
Oct 30 23:48:33.433970 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.433938    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-token\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.434187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.434090    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-data\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.434187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.434147    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-certs\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.434307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.434204    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-socket\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.434307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.434297    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-envoy\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.434442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.434388    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-podinfo\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.434499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.434454    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istiod-ca-cert\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.434647 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.434629    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-credential-socket\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.434706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.434660    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8qc\" (UniqueName: \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-kube-api-access-hz8qc\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.535897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.535864    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-credential-socket\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.535908    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8qc\" (UniqueName: \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-kube-api-access-hz8qc\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.535991    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-token\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.536121    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-data\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.536208    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-certs\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.536304    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-socket\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536538 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.536511    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-data\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.536400    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-credential-socket\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.536586    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-certs\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536733 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.536618    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-socket\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536792 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.536752    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-envoy\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.536786    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-podinfo\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.536852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.536818    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istiod-ca-cert\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.537697 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.537667    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istiod-ca-cert\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.539419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.539358    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-envoy\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.539999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.539977    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-podinfo\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.552171 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.552141    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-token\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.552827 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.552808    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8qc\" (UniqueName: \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-kube-api-access-hz8qc\") pod \"e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") " pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:33.601962 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:33.601671    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:34.213407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:34.213336    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"]
Oct 30 23:48:34.219415 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:48:34.216957    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76874aeb_95f4_483e_acca_cb1ba0b9ecfc.slice/crio-c9e257ed7f1b8e5aa5cfceb98f3145cab9bd1fa0a4ff3281a3ba2f7c6331b668 WatchSource:0}: Error finding container c9e257ed7f1b8e5aa5cfceb98f3145cab9bd1fa0a4ff3281a3ba2f7c6331b668: Status 404 returned error can't find the container with id c9e257ed7f1b8e5aa5cfceb98f3145cab9bd1fa0a4ff3281a3ba2f7c6331b668
Oct 30 23:48:34.219698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:34.219627    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:48:34.219698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:34.219692    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:48:34.219797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:34.219731    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:48:34.797589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:34.797550    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" event={"ID":"76874aeb-95f4-483e-acca-cb1ba0b9ecfc","Type":"ContainerStarted","Data":"03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01"}
Oct 30 23:48:34.798099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:34.797646    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" event={"ID":"76874aeb-95f4-483e-acca-cb1ba0b9ecfc","Type":"ContainerStarted","Data":"c9e257ed7f1b8e5aa5cfceb98f3145cab9bd1fa0a4ff3281a3ba2f7c6331b668"}
Oct 30 23:48:34.834251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:34.834198    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" podStartSLOduration=1.834185215 podStartE2EDuration="1.834185215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:34.830547795 +0000 UTC m=+1773.757185381" watchObservedRunningTime="2026-10-30 23:48:34.834185215 +0000 UTC m=+1773.760822796"
Oct 30 23:48:35.144589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:35.144494    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"]
Oct 30 23:48:35.147544 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:35.147511    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e33d7f-dz7t5"]
Oct 30 23:48:35.256040 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:35.255993    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f309e1fd-b984-4023-9a90-031a688fd4a8" path="/var/lib/kubelet/pods/f309e1fd-b984-4023-9a90-031a688fd4a8/volumes"
Oct 30 23:48:35.603334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:35.603291    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:35.611724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:35.611697    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:35.809443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:35.809412    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:35.810442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:35.810416    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:48:39.472436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.472400    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:48:39.543012 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.542979    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be309a26-4bd4-4f00-927c-f9b0fb48ca31-kube-api-access-n8vdp\" (UniqueName: \"kubernetes.io/projected/be309a26-4bd4-4f00-927c-f9b0fb48ca31-kube-api-access-n8vdp\") pod \"be309a26-4bd4-4f00-927c-f9b0fb48ca31\" (UID: \"be309a26-4bd4-4f00-927c-f9b0fb48ca31\") "
Oct 30 23:48:39.543231 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.543213    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be309a26-4bd4-4f00-927c-f9b0fb48ca31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be309a26-4bd4-4f00-927c-f9b0fb48ca31-plugins-config-volume\") pod \"be309a26-4bd4-4f00-927c-f9b0fb48ca31\" (UID: \"be309a26-4bd4-4f00-927c-f9b0fb48ca31\") "
Oct 30 23:48:39.543700 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.543648    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be309a26-4bd4-4f00-927c-f9b0fb48ca31-plugins-config-volume" pod "be309a26-4bd4-4f00-927c-f9b0fb48ca31" (UID: "be309a26-4bd4-4f00-927c-f9b0fb48ca31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:39.544301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.544277    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be309a26-4bd4-4f00-927c-f9b0fb48ca31-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:39.545695 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.545664    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be309a26-4bd4-4f00-927c-f9b0fb48ca31-kube-api-access-n8vdp" pod "be309a26-4bd4-4f00-927c-f9b0fb48ca31" (UID: "be309a26-4bd4-4f00-927c-f9b0fb48ca31"). InnerVolumeSpecName "kube-api-access-n8vdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:39.646179 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.646144    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8vdp\" (UniqueName: \"kubernetes.io/projected/be309a26-4bd4-4f00-927c-f9b0fb48ca31-kube-api-access-n8vdp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:39.930095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.930056    4555 generic.go:356] "Generic (PLEG): container finished" podID="be309a26-4bd4-4f00-927c-f9b0fb48ca31" containerID="cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62" exitCode=0
Oct 30 23:48:39.930285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.930116    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw" event={"ID":"be309a26-4bd4-4f00-927c-f9b0fb48ca31","Type":"ContainerDied","Data":"cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62"}
Oct 30 23:48:39.930285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.930170    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw" event={"ID":"be309a26-4bd4-4f00-927c-f9b0fb48ca31","Type":"ContainerDied","Data":"c94f7d159767f7e8ae0e3e6df20749aa7d8d31160b67b8038cdb9f4386d85b6a"}
Oct 30 23:48:39.930285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.930121    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"
Oct 30 23:48:39.930285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.930201    4555 scope.go:122] "RemoveContainer" containerID="cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62"
Oct 30 23:48:39.942084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.942066    4555 scope.go:122] "RemoveContainer" containerID="cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62"
Oct 30 23:48:39.942385 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:48:39.942347    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62\": container with ID starting with cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62 not found: ID does not exist" containerID="cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62"
Oct 30 23:48:39.942462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.942396    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62"} err="failed to get container status \"cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62\": rpc error: code = NotFound desc = could not find container \"cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62\": container with ID starting with cb4c91925457c68a42dd40e0e1f4b23d1f7ee08d8df2538cdaa4084578656d62 not found: ID does not exist"
Oct 30 23:48:39.969196 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.969154    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"]
Oct 30 23:48:39.974406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:39.974361    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a162557-5bf59d74bf-pdltw"]
Oct 30 23:48:40.860031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:40.860000    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"]
Oct 30 23:48:40.864176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:40.864152    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6cb71-6qvfz"]
Oct 30 23:48:41.209863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.209784    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366066ca-a3ff-4013-a7f0-c03daa16d502" path="/var/lib/kubelet/pods/366066ca-a3ff-4013-a7f0-c03daa16d502/volumes"
Oct 30 23:48:41.210269 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.210255    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be309a26-4bd4-4f00-927c-f9b0fb48ca31" path="/var/lib/kubelet/pods/be309a26-4bd4-4f00-927c-f9b0fb48ca31/volumes"
Oct 30 23:48:41.565224 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.565188    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"]
Oct 30 23:48:41.565914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.565897    4555 cpu_manager.go:414] "containerMap: removing container" podUID="be309a26-4bd4-4f00-927c-f9b0fb48ca31" containerName="payload-processing"
Oct 30 23:48:41.565914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.565913    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be309a26-4bd4-4f00-927c-f9b0fb48ca31" containerName="payload-processing"
Oct 30 23:48:41.566084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.566021    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="be309a26-4bd4-4f00-927c-f9b0fb48ca31" containerName="payload-processing"
Oct 30 23:48:41.571301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.571279    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:48:41.577782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.576058    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1bc837-dockercfg-r9rjj\""
Oct 30 23:48:41.582331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.582183    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"]
Oct 30 23:48:41.582537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.582344    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1bc837\""
Oct 30 23:48:41.721397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.720870    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxpf\" (UniqueName: \"kubernetes.io/projected/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-kube-api-access-9dxpf\") pod \"payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l\" (UID: \"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:48:41.721397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.721088    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l\" (UID: \"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:48:41.873446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.867020    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l\" (UID: \"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:48:41.873446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.867290    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxpf\" (UniqueName: \"kubernetes.io/projected/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-kube-api-access-9dxpf\") pod \"payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l\" (UID: \"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:48:41.873446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.868724    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l\" (UID: \"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:48:41.955399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:41.954911    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxpf\" (UniqueName: \"kubernetes.io/projected/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-kube-api-access-9dxpf\") pod \"payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l\" (UID: \"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:48:42.189860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:42.189631    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:48:42.278057 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:42.278013    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"]
Oct 30 23:48:42.278723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:42.278679    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" containerID="cri-o://8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f" gracePeriod=30
Oct 30 23:48:42.565345 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:42.564517    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"]
Oct 30 23:48:42.578198 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:48:42.577629    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87ec77e_7053_4a36_be37_9c7dd2fcbd8b.slice/crio-7849e90fea2a266eb50f089c1e14b1b4e1e9e34c1b749e048402e7c834f313a4 WatchSource:0}: Error finding container 7849e90fea2a266eb50f089c1e14b1b4e1e9e34c1b749e048402e7c834f313a4: Status 404 returned error can't find the container with id 7849e90fea2a266eb50f089c1e14b1b4e1e9e34c1b749e048402e7c834f313a4
Oct 30 23:48:43.034587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:43.034539    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l" event={"ID":"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b","Type":"ContainerStarted","Data":"486167aefb366ea9aea5af235913543f121c2455cfec874ff4ac991dc3a964ca"}
Oct 30 23:48:43.035001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:43.034607    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l" event={"ID":"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b","Type":"ContainerStarted","Data":"7849e90fea2a266eb50f089c1e14b1b4e1e9e34c1b749e048402e7c834f313a4"}
Oct 30 23:48:43.035001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:43.034640    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:48:43.058195 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:43.058140    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l" podStartSLOduration=2.058126345 podStartE2EDuration="2.058126345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:43.052770922 +0000 UTC m=+1781.979408503" watchObservedRunningTime="2026-10-30 23:48:43.058126345 +0000 UTC m=+1781.984764007"
Oct 30 23:48:43.127261 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:43.127225    4555 patch_prober.go:29] interesting pod/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:48:43.127466 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:43.127281    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:48:44.555142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:44.555107    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"]
Oct 30 23:48:44.559922 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:44.559895    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-dxklv"]
Oct 30 23:48:45.212130 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:45.212091    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412cbf1d-cb7a-452f-9435-41f3b7b5bec4" path="/var/lib/kubelet/pods/412cbf1d-cb7a-452f-9435-41f3b7b5bec4/volumes"
Oct 30 23:48:47.625101 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.625076    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:48:47.766364 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766326    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-certs\") pod \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") "
Oct 30 23:48:47.766575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766455    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-data\") pod \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") "
Oct 30 23:48:47.766575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766527    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-token\" (UniqueName: \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-token\") pod \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") "
Oct 30 23:48:47.766575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766558    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-credential-socket\") pod \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") "
Oct 30 23:48:47.766777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766581    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-envoy\") pod \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") "
Oct 30 23:48:47.766777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766636    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istiod-ca-cert\") pod \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") "
Oct 30 23:48:47.766777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766662    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-certs" pod "4d8379c8-e077-4d6a-a325-a91ce3c911bb" (UID: "4d8379c8-e077-4d6a-a325-a91ce3c911bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:48:47.766777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766725    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-socket\") pod \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") "
Oct 30 23:48:47.766777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766767    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-data" pod "4d8379c8-e077-4d6a-a325-a91ce3c911bb" (UID: "4d8379c8-e077-4d6a-a325-a91ce3c911bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:48:47.767047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766802    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-podinfo\") pod \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") "
Oct 30 23:48:47.767047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.766906    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-kube-api-access-z7xck\" (UniqueName: \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-kube-api-access-z7xck\") pod \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\" (UID: \"4d8379c8-e077-4d6a-a325-a91ce3c911bb\") "
Oct 30 23:48:47.767047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.767007    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-socket" pod "4d8379c8-e077-4d6a-a325-a91ce3c911bb" (UID: "4d8379c8-e077-4d6a-a325-a91ce3c911bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:48:47.767207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.767129    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istiod-ca-cert" pod "4d8379c8-e077-4d6a-a325-a91ce3c911bb" (UID: "4d8379c8-e077-4d6a-a325-a91ce3c911bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:47.767313 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.767293    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-credential-socket" pod "4d8379c8-e077-4d6a-a325-a91ce3c911bb" (UID: "4d8379c8-e077-4d6a-a325-a91ce3c911bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:48:47.767689 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.767661    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:47.767689 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.767689    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:47.767899 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.767706    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:47.767899 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.767719    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:47.767899 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.767731    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:47.769706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.769680    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-envoy" pod "4d8379c8-e077-4d6a-a325-a91ce3c911bb" (UID: "4d8379c8-e077-4d6a-a325-a91ce3c911bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:48:47.769842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.769732    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-podinfo" pod "4d8379c8-e077-4d6a-a325-a91ce3c911bb" (UID: "4d8379c8-e077-4d6a-a325-a91ce3c911bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:48:47.770824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.770802    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-token" pod "4d8379c8-e077-4d6a-a325-a91ce3c911bb" (UID: "4d8379c8-e077-4d6a-a325-a91ce3c911bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:47.770824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.770807    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-kube-api-access-z7xck" pod "4d8379c8-e077-4d6a-a325-a91ce3c911bb" (UID: "4d8379c8-e077-4d6a-a325-a91ce3c911bb"). InnerVolumeSpecName "kube-api-access-z7xck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:47.869730 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.869645    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:47.869730 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.869675    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:47.869730 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.869691    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8379c8-e077-4d6a-a325-a91ce3c911bb-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:47.869730 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:47.869701    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7xck\" (UniqueName: \"kubernetes.io/projected/4d8379c8-e077-4d6a-a325-a91ce3c911bb-kube-api-access-z7xck\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:48.186324 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:48.186240    4555 generic.go:356] "Generic (PLEG): container finished" podID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerID="8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f" exitCode=0
Oct 30 23:48:48.186577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:48.186554    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" event={"ID":"4d8379c8-e077-4d6a-a325-a91ce3c911bb","Type":"ContainerDied","Data":"8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f"}
Oct 30 23:48:48.186738 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:48.186719    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527" event={"ID":"4d8379c8-e077-4d6a-a325-a91ce3c911bb","Type":"ContainerDied","Data":"cddf4fc2af4503281c6acaddfe963f6f415fe6a2856970a0a36a7b870edd0424"}
Oct 30 23:48:48.186849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:48.186837    4555 scope.go:122] "RemoveContainer" containerID="8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f"
Oct 30 23:48:48.187109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:48.187094    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"
Oct 30 23:48:48.209468 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:48.209033    4555 scope.go:122] "RemoveContainer" containerID="8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f"
Oct 30 23:48:48.209557 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:48:48.209475    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f\": container with ID starting with 8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f not found: ID does not exist" containerID="8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f"
Oct 30 23:48:48.209557 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:48.209511    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f"} err="failed to get container status \"8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f\": rpc error: code = NotFound desc = could not find container \"8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f\": container with ID starting with 8a671515dca764957484c07e57a0061f0fa73bc45232e7f5994de9680d39531f not found: ID does not exist"
Oct 30 23:48:48.250354 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:48.250321    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"]
Oct 30 23:48:48.257994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:48.257968    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-2g527"]
Oct 30 23:48:49.211968 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:49.211934    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" path="/var/lib/kubelet/pods/4d8379c8-e077-4d6a-a325-a91ce3c911bb/volumes"
Oct 30 23:48:54.120542 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:48:54.120508    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:48:57.447760 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:48:57.447724    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:49:02.693848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:02.693784    4555 scope.go:122] "RemoveContainer" containerID="1266aa490793c0506a7bf8ecfe0f2e0d2cfafa1c74f931cb95657492145f2ee6"
Oct 30 23:49:02.705657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:02.705620    4555 scope.go:122] "RemoveContainer" containerID="cfdb409ad63052d14743813ba985299d72e882bad5b031baa4f80d5e83edf38e"
Oct 30 23:49:02.717068 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:02.717050    4555 scope.go:122] "RemoveContainer" containerID="eb08ffe1eede148e1713bcc713c6c17bffa4d00d7739701c1e1f6af048d74521"
Oct 30 23:49:02.726574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:02.726558    4555 scope.go:122] "RemoveContainer" containerID="01bd42e200c132c9fab68c36a651f0a20ad3a43dfd078314d67cd8a05d840ad0"
Oct 30 23:49:02.736108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:02.736087    4555 scope.go:122] "RemoveContainer" containerID="8dc3435e08f3c807b806278f3e044e2890f129aacd57d1ee8b7846f8904ed465"
Oct 30 23:49:02.746570 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:02.746552    4555 scope.go:122] "RemoveContainer" containerID="85d131cdfb2af6aac94abb97dd6edda4d049a10216ea0b5e2e8b1dd69e53b6f5"
Oct 30 23:49:11.738043 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.738005    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"]
Oct 30 23:49:11.739482 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.739444    4555 cpu_manager.go:414] "containerMap: removing container" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy"
Oct 30 23:49:11.739482 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.739481    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy"
Oct 30 23:49:11.739742 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.739726    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d8379c8-e077-4d6a-a325-a91ce3c911bb" containerName="istio-proxy"
Oct 30 23:49:11.758393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.750664    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.759660 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.759628    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"]
Oct 30 23:49:11.765361 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.763729    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6a162557-openshift-default-dockercfg-w6bzl\""
Oct 30 23:49:11.789427 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.766772    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"]
Oct 30 23:49:11.789427 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.767061    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" containerID="cri-o://1bbbaa93df5f304d4353779b02aeba208f816147f6038b069025749ba2d81ca5" gracePeriod=30
Oct 30 23:49:11.815453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.814472    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-data\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.815453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.814569    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.815453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.814619    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-certs\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.815453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.814655    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-podinfo\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.815453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.814719    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a6da0-1746-45de-808f-e9a8b3a26ce3-istiod-ca-cert\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.815453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.814777    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-credential-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.815453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.814809    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9bj\" (UniqueName: \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-kube-api-access-8w9bj\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.815453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.814870    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-envoy\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.815453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.814950    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-token\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.915905 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.915874    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-token\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.915985    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-data\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916037    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916079    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-certs\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916102    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-podinfo\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916163    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a6da0-1746-45de-808f-e9a8b3a26ce3-istiod-ca-cert\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916237    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-credential-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916558 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916275    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9bj\" (UniqueName: \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-kube-api-access-8w9bj\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916558 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916346    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-envoy\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916594    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916760 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916726    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-data\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916809 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916752    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-certs\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916875 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916820    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-credential-socket\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.916930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.916912    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a6da0-1746-45de-808f-e9a8b3a26ce3-istiod-ca-cert\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.918844 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.918816    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-envoy\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.919159 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.919139    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-podinfo\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.928701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.928677    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-token\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:11.928800 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:11.928764    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9bj\" (UniqueName: \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-kube-api-access-8w9bj\") pod \"e2e-mt-6a162557-openshift-default-8677879c89-nprf6\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") " pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:12.100263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:12.100213    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:12.503468 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:12.501521    4555 patch_prober.go:29] interesting pod/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:49:12.503468 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:12.501581    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:49:12.825488 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:49:12.825448    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277a6da0_1746_45de_808f_e9a8b3a26ce3.slice/crio-4afcf0155f61f4e2bb6bc30a420147d296f931a79380ba64f966db57daaf7651 WatchSource:0}: Error finding container 4afcf0155f61f4e2bb6bc30a420147d296f931a79380ba64f966db57daaf7651: Status 404 returned error can't find the container with id 4afcf0155f61f4e2bb6bc30a420147d296f931a79380ba64f966db57daaf7651
Oct 30 23:49:12.827295 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:12.827266    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"]
Oct 30 23:49:12.828283 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:12.828253    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:49:12.828345 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:12.828318    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:49:12.828548 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:12.828349    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:49:13.614155 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:13.614106    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" event={"ID":"277a6da0-1746-45de-808f-e9a8b3a26ce3","Type":"ContainerStarted","Data":"8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e"}
Oct 30 23:49:13.614338 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:13.614213    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" event={"ID":"277a6da0-1746-45de-808f-e9a8b3a26ce3","Type":"ContainerStarted","Data":"4afcf0155f61f4e2bb6bc30a420147d296f931a79380ba64f966db57daaf7651"}
Oct 30 23:49:13.644357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:13.644306    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" podStartSLOduration=2.644290317 podStartE2EDuration="2.644290317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:13.639404963 +0000 UTC m=+1812.566042545" watchObservedRunningTime="2026-10-30 23:49:13.644290317 +0000 UTC m=+1812.570927898"
Oct 30 23:49:14.101405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:14.101351    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:14.102959 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:14.102933    4555 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-nprf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 23:49:14.103095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:14.102987    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 23:49:15.100910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:15.100876    4555 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-nprf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 23:49:15.101110 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:15.100926    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 23:49:16.117889 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:16.117847    4555 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-nprf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 23:49:16.120239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:16.120204    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 23:49:17.101315 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.100885    4555 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-nprf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 23:49:17.101315 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.100939    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 23:49:17.713839 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.713764    4555 generic.go:356] "Generic (PLEG): container finished" podID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerID="1bbbaa93df5f304d4353779b02aeba208f816147f6038b069025749ba2d81ca5" exitCode=0
Oct 30 23:49:17.714203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.713840    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" event={"ID":"5625facc-0053-4711-ae40-3e0f5d9fabf8","Type":"ContainerDied","Data":"1bbbaa93df5f304d4353779b02aeba208f816147f6038b069025749ba2d81ca5"}
Oct 30 23:49:17.714203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.713901    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm" event={"ID":"5625facc-0053-4711-ae40-3e0f5d9fabf8","Type":"ContainerDied","Data":"039a8fb3a6c380829840ddefd15ecfbe4d934af51cb484142fb52e062709e904"}
Oct 30 23:49:17.714203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.713927    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="039a8fb3a6c380829840ddefd15ecfbe4d934af51cb484142fb52e062709e904"
Oct 30 23:49:17.715582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.715565    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:49:17.877397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.865901    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-socket\") pod \"5625facc-0053-4711-ae40-3e0f5d9fabf8\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") "
Oct 30 23:49:17.877854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.877823    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-envoy\") pod \"5625facc-0053-4711-ae40-3e0f5d9fabf8\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") "
Oct 30 23:49:17.878058 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.878041    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5625facc-0053-4711-ae40-3e0f5d9fabf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5625facc-0053-4711-ae40-3e0f5d9fabf8-istiod-ca-cert\") pod \"5625facc-0053-4711-ae40-3e0f5d9fabf8\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") "
Oct 30 23:49:17.878200 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.878187    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-credential-socket\") pod \"5625facc-0053-4711-ae40-3e0f5d9fabf8\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") "
Oct 30 23:49:17.878335 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.878321    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-certs\") pod \"5625facc-0053-4711-ae40-3e0f5d9fabf8\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") "
Oct 30 23:49:17.878511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.878499    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-kube-api-access-jrvhf\" (UniqueName: \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-kube-api-access-jrvhf\") pod \"5625facc-0053-4711-ae40-3e0f5d9fabf8\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") "
Oct 30 23:49:17.878635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.878626    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-podinfo\") pod \"5625facc-0053-4711-ae40-3e0f5d9fabf8\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") "
Oct 30 23:49:17.878706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.878698    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-data\") pod \"5625facc-0053-4711-ae40-3e0f5d9fabf8\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") "
Oct 30 23:49:17.878790 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.878779    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-token\" (UniqueName: \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-token\") pod \"5625facc-0053-4711-ae40-3e0f5d9fabf8\" (UID: \"5625facc-0053-4711-ae40-3e0f5d9fabf8\") "
Oct 30 23:49:17.880539 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.880507    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-credential-socket" pod "5625facc-0053-4711-ae40-3e0f5d9fabf8" (UID: "5625facc-0053-4711-ae40-3e0f5d9fabf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:17.880693 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.880673    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-socket" pod "5625facc-0053-4711-ae40-3e0f5d9fabf8" (UID: "5625facc-0053-4711-ae40-3e0f5d9fabf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:17.880768 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.880714    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-data" pod "5625facc-0053-4711-ae40-3e0f5d9fabf8" (UID: "5625facc-0053-4711-ae40-3e0f5d9fabf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:17.881128 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.881102    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5625facc-0053-4711-ae40-3e0f5d9fabf8-istiod-ca-cert" pod "5625facc-0053-4711-ae40-3e0f5d9fabf8" (UID: "5625facc-0053-4711-ae40-3e0f5d9fabf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:49:17.882549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.882521    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-certs" pod "5625facc-0053-4711-ae40-3e0f5d9fabf8" (UID: "5625facc-0053-4711-ae40-3e0f5d9fabf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:17.899038 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.898975    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-podinfo" pod "5625facc-0053-4711-ae40-3e0f5d9fabf8" (UID: "5625facc-0053-4711-ae40-3e0f5d9fabf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:49:17.899429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.899347    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-token" pod "5625facc-0053-4711-ae40-3e0f5d9fabf8" (UID: "5625facc-0053-4711-ae40-3e0f5d9fabf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:17.901098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.901057    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-envoy" pod "5625facc-0053-4711-ae40-3e0f5d9fabf8" (UID: "5625facc-0053-4711-ae40-3e0f5d9fabf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:17.902946 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.902909    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-kube-api-access-jrvhf" pod "5625facc-0053-4711-ae40-3e0f5d9fabf8" (UID: "5625facc-0053-4711-ae40-3e0f5d9fabf8"). InnerVolumeSpecName "kube-api-access-jrvhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:17.981305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.980990    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:17.981305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.981045    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:17.981305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.981067    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:17.981305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.981086    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5625facc-0053-4711-ae40-3e0f5d9fabf8-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:17.981305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.981108    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:17.981305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.981128    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:17.981305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.981150    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrvhf\" (UniqueName: \"kubernetes.io/projected/5625facc-0053-4711-ae40-3e0f5d9fabf8-kube-api-access-jrvhf\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:17.981305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.981173    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:17.981305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:17.981191    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5625facc-0053-4711-ae40-3e0f5d9fabf8-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:18.102396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.101361    4555 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-nprf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 23:49:18.102396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.101459    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 23:49:18.736176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.736142    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"
Oct 30 23:49:18.777394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.776670    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"]
Oct 30 23:49:18.784406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.784338    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-2dpbm"]
Oct 30 23:49:18.955547 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.955498    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11064-aa12de"]
Oct 30 23:49:18.956909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.956886    4555 cpu_manager.go:414] "containerMap: removing container" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy"
Oct 30 23:49:18.957089 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.957076    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy"
Oct 30 23:49:18.957416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.957400    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" containerName="istio-proxy"
Oct 30 23:49:18.973748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.973725    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-aa12de"
Oct 30 23:49:18.979671 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.979150    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-csqxw\""
Oct 30 23:49:18.985982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:18.985923    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11064-aa12de"]
Oct 30 23:49:19.134253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:19.134209    4555 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-nprf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 23:49:19.134583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:19.134549    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 23:49:19.209395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:19.190888    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6cz4\" (UniqueName: \"kubernetes.io/projected/995b8fd8-49bd-46cd-908a-f1a035159e02-kube-api-access-n6cz4\") pod \"test-curl-11064-aa12de\" (UID: \"995b8fd8-49bd-46cd-908a-f1a035159e02\") " pod="opendatahub/test-curl-11064-aa12de"
Oct 30 23:49:19.329235 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:19.328690    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5625facc-0053-4711-ae40-3e0f5d9fabf8" path="/var/lib/kubelet/pods/5625facc-0053-4711-ae40-3e0f5d9fabf8/volumes"
Oct 30 23:49:19.330951 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:19.330428    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6cz4\" (UniqueName: \"kubernetes.io/projected/995b8fd8-49bd-46cd-908a-f1a035159e02-kube-api-access-n6cz4\") pod \"test-curl-11064-aa12de\" (UID: \"995b8fd8-49bd-46cd-908a-f1a035159e02\") " pod="opendatahub/test-curl-11064-aa12de"
Oct 30 23:49:19.346253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:19.346225    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6cz4\" (UniqueName: \"kubernetes.io/projected/995b8fd8-49bd-46cd-908a-f1a035159e02-kube-api-access-n6cz4\") pod \"test-curl-11064-aa12de\" (UID: \"995b8fd8-49bd-46cd-908a-f1a035159e02\") " pod="opendatahub/test-curl-11064-aa12de"
Oct 30 23:49:19.616760 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:19.616679    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-aa12de"
Oct 30 23:49:19.956443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:19.955693    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11064-aa12de"]
Oct 30 23:49:20.113078 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:20.113031    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:20.114508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:20.114473    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:20.117176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:20.117094    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:49:20.767047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:20.766939    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11064-aa12de" event={"ID":"995b8fd8-49bd-46cd-908a-f1a035159e02","Type":"ContainerStarted","Data":"4c061cdbd00fdce32f80897a389e258f7b2e12190f0b5cecf989bd8ce7bf9f95"}
Oct 30 23:49:20.767047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:20.766989    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11064-aa12de" event={"ID":"995b8fd8-49bd-46cd-908a-f1a035159e02","Type":"ContainerStarted","Data":"3686e4b55ecdc454948f18863c64bb533eedb8ac05c84b8e69aa78c28ba7e767"}
Oct 30 23:49:20.787807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:20.787744    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11064-aa12de" podStartSLOduration=2.787729081 podStartE2EDuration="2.787729081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:20.783410009 +0000 UTC m=+1819.710047588" watchObservedRunningTime="2026-10-30 23:49:20.787729081 +0000 UTC m=+1819.714366661"
Oct 30 23:49:21.791475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:21.789988    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11064-aa12de"]
Oct 30 23:49:21.944096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:21.926747    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11064-aa12de"]
Oct 30 23:49:21.944096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:21.935488    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11064-5624c6"]
Oct 30 23:49:21.944096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:21.936687    4555 cpu_manager.go:414] "containerMap: removing container" podUID="995b8fd8-49bd-46cd-908a-f1a035159e02" containerName="test-curl-11064-aa12de"
Oct 30 23:49:21.944096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:21.936709    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995b8fd8-49bd-46cd-908a-f1a035159e02" containerName="test-curl-11064-aa12de"
Oct 30 23:49:21.944096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:21.936957    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="995b8fd8-49bd-46cd-908a-f1a035159e02" containerName="test-curl-11064-aa12de"
Oct 30 23:49:21.944096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:21.941418    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:21.944096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:21.942906    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11064-5624c6"]
Oct 30 23:49:22.094490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:22.094362    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4d9k\" (UniqueName: \"kubernetes.io/projected/6bac5dd3-132a-44fc-bf66-c4ed54596584-kube-api-access-c4d9k\") pod \"test-curl-11064-5624c6\" (UID: \"6bac5dd3-132a-44fc-bf66-c4ed54596584\") " pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:22.196684 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:22.196637    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4d9k\" (UniqueName: \"kubernetes.io/projected/6bac5dd3-132a-44fc-bf66-c4ed54596584-kube-api-access-c4d9k\") pod \"test-curl-11064-5624c6\" (UID: \"6bac5dd3-132a-44fc-bf66-c4ed54596584\") " pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:22.209007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:22.208975    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4d9k\" (UniqueName: \"kubernetes.io/projected/6bac5dd3-132a-44fc-bf66-c4ed54596584-kube-api-access-c4d9k\") pod \"test-curl-11064-5624c6\" (UID: \"6bac5dd3-132a-44fc-bf66-c4ed54596584\") " pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:22.257583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:22.257546    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:22.547439 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:49:22.547409    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bac5dd3_132a_44fc_bf66_c4ed54596584.slice/crio-4ab356243882933fba502dc5e4d17a341255f992cd5093b40ac9efcf4c0390bd WatchSource:0}: Error finding container 4ab356243882933fba502dc5e4d17a341255f992cd5093b40ac9efcf4c0390bd: Status 404 returned error can't find the container with id 4ab356243882933fba502dc5e4d17a341255f992cd5093b40ac9efcf4c0390bd
Oct 30 23:49:22.548162 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:22.548051    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11064-5624c6"]
Oct 30 23:49:22.859454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:22.859351    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11064-5624c6" event={"ID":"6bac5dd3-132a-44fc-bf66-c4ed54596584","Type":"ContainerStarted","Data":"c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126"}
Oct 30 23:49:22.859454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:22.859419    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11064-5624c6" event={"ID":"6bac5dd3-132a-44fc-bf66-c4ed54596584","Type":"ContainerStarted","Data":"4ab356243882933fba502dc5e4d17a341255f992cd5093b40ac9efcf4c0390bd"}
Oct 30 23:49:22.859919 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:22.859603    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11064-aa12de" podUID="995b8fd8-49bd-46cd-908a-f1a035159e02" containerName="test-curl-11064-aa12de" containerID="cri-o://4c061cdbd00fdce32f80897a389e258f7b2e12190f0b5cecf989bd8ce7bf9f95" gracePeriod=2
Oct 30 23:49:22.877537 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:22.877497    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-aa12de\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="995b8fd8-49bd-46cd-908a-f1a035159e02" pod="opendatahub/test-curl-11064-aa12de"
Oct 30 23:49:22.881439 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:22.881382    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11064-5624c6" podStartSLOduration=1.881346868 podStartE2EDuration="1.881346868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:22.874825334 +0000 UTC m=+1821.801462916" watchObservedRunningTime="2026-10-30 23:49:22.881346868 +0000 UTC m=+1821.807984450"
Oct 30 23:49:23.707390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:23.707226    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11064-5624c6"]
Oct 30 23:49:23.724443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:23.723620    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11064-5624c6"]
Oct 30 23:49:23.844664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:23.844631    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11064-86556d"]
Oct 30 23:49:23.845340 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:23.845325    4555 cpu_manager.go:414] "containerMap: removing container" podUID="6bac5dd3-132a-44fc-bf66-c4ed54596584" containerName="test-curl-11064-5624c6"
Oct 30 23:49:23.845456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:23.845342    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bac5dd3-132a-44fc-bf66-c4ed54596584" containerName="test-curl-11064-5624c6"
Oct 30 23:49:23.845456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:23.845449    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bac5dd3-132a-44fc-bf66-c4ed54596584" containerName="test-curl-11064-5624c6"
Oct 30 23:49:23.849008 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:23.848988    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-86556d"
Oct 30 23:49:23.857793 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:23.857769    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11064-86556d"]
Oct 30 23:49:23.938699 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:23.938670    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzqc\" (UniqueName: \"kubernetes.io/projected/269dc58d-0b23-4f44-9f73-3910c85a3cfb-kube-api-access-7zzqc\") pod \"test-curl-11064-86556d\" (UID: \"269dc58d-0b23-4f44-9f73-3910c85a3cfb\") " pod="opendatahub/test-curl-11064-86556d"
Oct 30 23:49:24.040790 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:24.040694    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzqc\" (UniqueName: \"kubernetes.io/projected/269dc58d-0b23-4f44-9f73-3910c85a3cfb-kube-api-access-7zzqc\") pod \"test-curl-11064-86556d\" (UID: \"269dc58d-0b23-4f44-9f73-3910c85a3cfb\") " pod="opendatahub/test-curl-11064-86556d"
Oct 30 23:49:24.054230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:24.054194    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzqc\" (UniqueName: \"kubernetes.io/projected/269dc58d-0b23-4f44-9f73-3910c85a3cfb-kube-api-access-7zzqc\") pod \"test-curl-11064-86556d\" (UID: \"269dc58d-0b23-4f44-9f73-3910c85a3cfb\") " pod="opendatahub/test-curl-11064-86556d"
Oct 30 23:49:24.172635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:24.172600    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-86556d"
Oct 30 23:49:24.496284 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:49:24.496237    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod269dc58d_0b23_4f44_9f73_3910c85a3cfb.slice/crio-5f72a7635f30487dc69e05beef33ed171acee5be9714aa965f941df0d5884f79 WatchSource:0}: Error finding container 5f72a7635f30487dc69e05beef33ed171acee5be9714aa965f941df0d5884f79: Status 404 returned error can't find the container with id 5f72a7635f30487dc69e05beef33ed171acee5be9714aa965f941df0d5884f79
Oct 30 23:49:24.496671 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:24.496634    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11064-86556d"]
Oct 30 23:49:24.891641 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:24.891604    4555 generic.go:356] "Generic (PLEG): container finished" podID="995b8fd8-49bd-46cd-908a-f1a035159e02" containerID="4c061cdbd00fdce32f80897a389e258f7b2e12190f0b5cecf989bd8ce7bf9f95" exitCode=137
Oct 30 23:49:24.893902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:24.893637    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11064-86556d" event={"ID":"269dc58d-0b23-4f44-9f73-3910c85a3cfb","Type":"ContainerStarted","Data":"6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367"}
Oct 30 23:49:24.893902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:24.893684    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11064-86556d" event={"ID":"269dc58d-0b23-4f44-9f73-3910c85a3cfb","Type":"ContainerStarted","Data":"5f72a7635f30487dc69e05beef33ed171acee5be9714aa965f941df0d5884f79"}
Oct 30 23:49:24.894082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:24.894006    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11064-5624c6" podUID="6bac5dd3-132a-44fc-bf66-c4ed54596584" containerName="test-curl-11064-5624c6" containerID="cri-o://c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126" gracePeriod=2
Oct 30 23:49:24.996407 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:24.996326    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-5624c6\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="6bac5dd3-132a-44fc-bf66-c4ed54596584" pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:24.996896 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:24.996833    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11064-86556d" podStartSLOduration=1.99681556 podStartE2EDuration="1.99681556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:24.961077623 +0000 UTC m=+1823.887715205" watchObservedRunningTime="2026-10-30 23:49:24.99681556 +0000 UTC m=+1823.923453143"
Oct 30 23:49:25.044925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.044900    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-aa12de"
Oct 30 23:49:25.048916 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:25.048884    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-5624c6\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="6bac5dd3-132a-44fc-bf66-c4ed54596584" pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:25.161892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.161809    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995b8fd8-49bd-46cd-908a-f1a035159e02-kube-api-access-n6cz4\" (UniqueName: \"kubernetes.io/projected/995b8fd8-49bd-46cd-908a-f1a035159e02-kube-api-access-n6cz4\") pod \"995b8fd8-49bd-46cd-908a-f1a035159e02\" (UID: \"995b8fd8-49bd-46cd-908a-f1a035159e02\") "
Oct 30 23:49:25.164200 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.164164    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995b8fd8-49bd-46cd-908a-f1a035159e02-kube-api-access-n6cz4" pod "995b8fd8-49bd-46cd-908a-f1a035159e02" (UID: "995b8fd8-49bd-46cd-908a-f1a035159e02"). InnerVolumeSpecName "kube-api-access-n6cz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:25.213359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.213322    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995b8fd8-49bd-46cd-908a-f1a035159e02" path="/var/lib/kubelet/pods/995b8fd8-49bd-46cd-908a-f1a035159e02/volumes"
Oct 30 23:49:25.273528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.273490    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6cz4\" (UniqueName: \"kubernetes.io/projected/995b8fd8-49bd-46cd-908a-f1a035159e02-kube-api-access-n6cz4\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:25.784433 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.778057    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"]
Oct 30 23:49:25.812393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.801538    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:25.836616 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.815801    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2ccd42\"/\"openshift-service-ca.crt\""
Oct 30 23:49:25.836616 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.819791    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2ccd42\"/\"default-dockercfg-jtt7d\""
Oct 30 23:49:25.836616 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.819908    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2ccd42\"/\"kube-root-ca.crt\""
Oct 30 23:49:25.836616 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.820001    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2ccd42\"/\"test-model-2ccd42-kserve-self-signed-certs\""
Oct 30 23:49:25.837992 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.837954    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-model-cache\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:25.838273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.838255    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-tmp-dir\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:25.838569 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.838548    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6whmc\" (UniqueName: \"kubernetes.io/projected/2490e605-01fd-4ce8-8911-611c95d43bd4-kube-api-access-6whmc\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:25.838728 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.838713    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2490e605-01fd-4ce8-8911-611c95d43bd4-tls-certs\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:25.843393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.838905    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-home\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:25.843676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:25.843654    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-dshm\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.022729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.012299    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-home\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.022729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.012423    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-dshm\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.022729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.012624    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-model-cache\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.022729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.012737    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-tmp-dir\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.022729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.012842    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6whmc\" (UniqueName: \"kubernetes.io/projected/2490e605-01fd-4ce8-8911-611c95d43bd4-kube-api-access-6whmc\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.022729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.012875    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2490e605-01fd-4ce8-8911-611c95d43bd4-tls-certs\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.022729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.014223    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-model-cache\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.022729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.014506    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-home\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.022729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.016762    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-tmp-dir\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.029006 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.028935    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"]
Oct 30 23:49:26.067707 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.067620    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-dshm\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.079393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.072105    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6whmc\" (UniqueName: \"kubernetes.io/projected/2490e605-01fd-4ce8-8911-611c95d43bd4-kube-api-access-6whmc\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.302395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.301538    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2490e605-01fd-4ce8-8911-611c95d43bd4-tls-certs\") pod \"test-model-2ccd42-kserve-6856ff8f9f-xgl64\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") " pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.400598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.382805    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-aa12de"
Oct 30 23:49:26.400598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.382999    4555 scope.go:122] "RemoveContainer" containerID="4c061cdbd00fdce32f80897a389e258f7b2e12190f0b5cecf989bd8ce7bf9f95"
Oct 30 23:49:26.479501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.479472    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11064-86556d"]
Oct 30 23:49:26.587144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.579056    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:26.738913 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.738834    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11064-86556d"]
Oct 30 23:49:26.760166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.754227    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11064-914b2c"]
Oct 30 23:49:26.760166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.755259    4555 cpu_manager.go:414] "containerMap: removing container" podUID="269dc58d-0b23-4f44-9f73-3910c85a3cfb" containerName="test-curl-11064-86556d"
Oct 30 23:49:26.760166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.755281    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269dc58d-0b23-4f44-9f73-3910c85a3cfb" containerName="test-curl-11064-86556d"
Oct 30 23:49:26.760166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.755547    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="269dc58d-0b23-4f44-9f73-3910c85a3cfb" containerName="test-curl-11064-86556d"
Oct 30 23:49:26.769424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.765118    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:26.769585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.769511    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11064-914b2c"]
Oct 30 23:49:26.916390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:26.914189    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8fwl\" (UniqueName: \"kubernetes.io/projected/01b3bbfc-856e-446c-a099-8b36d99e0525-kube-api-access-h8fwl\") pod \"test-curl-11064-914b2c\" (UID: \"01b3bbfc-856e-446c-a099-8b36d99e0525\") " pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:27.031578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.020460    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fwl\" (UniqueName: \"kubernetes.io/projected/01b3bbfc-856e-446c-a099-8b36d99e0525-kube-api-access-h8fwl\") pod \"test-curl-11064-914b2c\" (UID: \"01b3bbfc-856e-446c-a099-8b36d99e0525\") " pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:27.052983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.039073    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8fwl\" (UniqueName: \"kubernetes.io/projected/01b3bbfc-856e-446c-a099-8b36d99e0525-kube-api-access-h8fwl\") pod \"test-curl-11064-914b2c\" (UID: \"01b3bbfc-856e-446c-a099-8b36d99e0525\") " pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:27.133212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.133171    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:27.255035 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.255008    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"]
Oct 30 23:49:27.271724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.271703    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:27.329534 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.329506    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bac5dd3-132a-44fc-bf66-c4ed54596584-kube-api-access-c4d9k\" (UniqueName: \"kubernetes.io/projected/6bac5dd3-132a-44fc-bf66-c4ed54596584-kube-api-access-c4d9k\") pod \"6bac5dd3-132a-44fc-bf66-c4ed54596584\" (UID: \"6bac5dd3-132a-44fc-bf66-c4ed54596584\") "
Oct 30 23:49:27.331855 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.331827    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bac5dd3-132a-44fc-bf66-c4ed54596584-kube-api-access-c4d9k" pod "6bac5dd3-132a-44fc-bf66-c4ed54596584" (UID: "6bac5dd3-132a-44fc-bf66-c4ed54596584"). InnerVolumeSpecName "kube-api-access-c4d9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:27.397001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.396969    4555 generic.go:356] "Generic (PLEG): container finished" podID="6bac5dd3-132a-44fc-bf66-c4ed54596584" containerID="c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126" exitCode=137
Oct 30 23:49:27.397191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.397043    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:27.397191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.397062    4555 scope.go:122] "RemoveContainer" containerID="c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126"
Oct 30 23:49:27.399358 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.399323    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11064-86556d" podUID="269dc58d-0b23-4f44-9f73-3910c85a3cfb" containerName="test-curl-11064-86556d" containerID="cri-o://6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367" gracePeriod=2
Oct 30 23:49:27.400571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.400541    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" event={"ID":"2490e605-01fd-4ce8-8911-611c95d43bd4","Type":"ContainerStarted","Data":"4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad"}
Oct 30 23:49:27.400708 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.400583    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" event={"ID":"2490e605-01fd-4ce8-8911-611c95d43bd4","Type":"ContainerStarted","Data":"4a3708ceebecba4f9b47782337fbff85f24f8ef3ed621bb0356d7b4b26ba4f32"}
Oct 30 23:49:27.410111 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.410089    4555 scope.go:122] "RemoveContainer" containerID="c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126"
Oct 30 23:49:27.410546 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:27.410504    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126\": container with ID starting with c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126 not found: ID does not exist" containerID="c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126"
Oct 30 23:49:27.410654 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.410557    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126"} err="failed to get container status \"c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126\": rpc error: code = NotFound desc = could not find container \"c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126\": container with ID starting with c99c920936dd7f88862da295c336d804deac6adfc7882f2391bea4122aa41126 not found: ID does not exist"
Oct 30 23:49:27.423205 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.423153    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" podStartSLOduration=2.423135243 podStartE2EDuration="2.423135243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:27.420252233 +0000 UTC m=+1826.346889816" watchObservedRunningTime="2026-10-30 23:49:27.423135243 +0000 UTC m=+1826.349772826"
Oct 30 23:49:27.428172 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:27.427167    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-5624c6\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="6bac5dd3-132a-44fc-bf66-c4ed54596584" pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:27.431802 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.431773    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4d9k\" (UniqueName: \"kubernetes.io/projected/6bac5dd3-132a-44fc-bf66-c4ed54596584-kube-api-access-c4d9k\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:27.455574 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:27.455539    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:49:27.459399 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.459356    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11064-914b2c"]
Oct 30 23:49:27.460199 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:49:27.460178    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01b3bbfc_856e_446c_a099_8b36d99e0525.slice/crio-fb5af86937e5c6631a0b87aad90f288edb65df4da5e678d4f556b69ae3cbd2e6 WatchSource:0}: Error finding container fb5af86937e5c6631a0b87aad90f288edb65df4da5e678d4f556b69ae3cbd2e6: Status 404 returned error can't find the container with id fb5af86937e5c6631a0b87aad90f288edb65df4da5e678d4f556b69ae3cbd2e6
Oct 30 23:49:27.598440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:27.583138    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:28.413946 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:28.413841    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11064-914b2c" event={"ID":"01b3bbfc-856e-446c-a099-8b36d99e0525","Type":"ContainerStarted","Data":"946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b"}
Oct 30 23:49:28.413946 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:28.413905    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11064-914b2c" event={"ID":"01b3bbfc-856e-446c-a099-8b36d99e0525","Type":"ContainerStarted","Data":"fb5af86937e5c6631a0b87aad90f288edb65df4da5e678d4f556b69ae3cbd2e6"}
Oct 30 23:49:28.416644 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:28.416611    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-5624c6\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="6bac5dd3-132a-44fc-bf66-c4ed54596584" pod="opendatahub/test-curl-11064-5624c6"
Oct 30 23:49:28.435849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:28.435800    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11064-914b2c" podStartSLOduration=2.435787424 podStartE2EDuration="2.435787424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:28.43188891 +0000 UTC m=+1827.358526493" watchObservedRunningTime="2026-10-30 23:49:28.435787424 +0000 UTC m=+1827.362425005"
Oct 30 23:49:29.212252 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:29.212210    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bac5dd3-132a-44fc-bf66-c4ed54596584" path="/var/lib/kubelet/pods/6bac5dd3-132a-44fc-bf66-c4ed54596584/volumes"
Oct 30 23:49:29.212649 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:29.212631    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11064-914b2c"]
Oct 30 23:49:29.221309 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:29.221270    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11064-914b2c"]
Oct 30 23:49:29.616428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:29.616402    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-86556d"
Oct 30 23:49:29.719181 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:29.719147    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269dc58d-0b23-4f44-9f73-3910c85a3cfb-kube-api-access-7zzqc\" (UniqueName: \"kubernetes.io/projected/269dc58d-0b23-4f44-9f73-3910c85a3cfb-kube-api-access-7zzqc\") pod \"269dc58d-0b23-4f44-9f73-3910c85a3cfb\" (UID: \"269dc58d-0b23-4f44-9f73-3910c85a3cfb\") "
Oct 30 23:49:29.721383 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:29.721349    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269dc58d-0b23-4f44-9f73-3910c85a3cfb-kube-api-access-7zzqc" pod "269dc58d-0b23-4f44-9f73-3910c85a3cfb" (UID: "269dc58d-0b23-4f44-9f73-3910c85a3cfb"). InnerVolumeSpecName "kube-api-access-7zzqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:29.821230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:29.821191    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zzqc\" (UniqueName: \"kubernetes.io/projected/269dc58d-0b23-4f44-9f73-3910c85a3cfb-kube-api-access-7zzqc\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:30.507414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.500579    4555 generic.go:356] "Generic (PLEG): container finished" podID="269dc58d-0b23-4f44-9f73-3910c85a3cfb" containerID="6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367" exitCode=137
Oct 30 23:49:30.507414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.500844    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11064-914b2c" podUID="01b3bbfc-856e-446c-a099-8b36d99e0525" containerName="test-curl-11064-914b2c" containerID="cri-o://946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b" gracePeriod=2
Oct 30 23:49:30.507414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.500989    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-86556d"
Oct 30 23:49:30.507414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.501779    4555 scope.go:122] "RemoveContainer" containerID="6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367"
Oct 30 23:49:30.507414 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:30.503978    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-914b2c\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="01b3bbfc-856e-446c-a099-8b36d99e0525" pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:30.513550 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:30.509536    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-914b2c\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="01b3bbfc-856e-446c-a099-8b36d99e0525" pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:30.654391 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:30.653787    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-86556d\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="269dc58d-0b23-4f44-9f73-3910c85a3cfb" pod="opendatahub/test-curl-11064-86556d"
Oct 30 23:49:30.676397 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:30.656422    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-914b2c\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="01b3bbfc-856e-446c-a099-8b36d99e0525" pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:30.676397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.657088    4555 scope.go:122] "RemoveContainer" containerID="6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367"
Oct 30 23:49:30.676397 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:30.668793    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367\": container with ID starting with 6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367 not found: ID does not exist" containerID="6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367"
Oct 30 23:49:30.676397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.668840    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367"} err="failed to get container status \"6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367\": rpc error: code = NotFound desc = could not find container \"6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367\": container with ID starting with 6ffea508ef92940ad4eac9ba1564a04ed45a77a60d6db8b5aeb9f3388788d367 not found: ID does not exist"
Oct 30 23:49:30.734405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.730824    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11079-163a14"]
Oct 30 23:49:30.734405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.731922    4555 cpu_manager.go:414] "containerMap: removing container" podUID="01b3bbfc-856e-446c-a099-8b36d99e0525" containerName="test-curl-11064-914b2c"
Oct 30 23:49:30.734405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.731946    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b3bbfc-856e-446c-a099-8b36d99e0525" containerName="test-curl-11064-914b2c"
Oct 30 23:49:30.734405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.732220    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b3bbfc-856e-446c-a099-8b36d99e0525" containerName="test-curl-11064-914b2c"
Oct 30 23:49:30.755610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.755583    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-163a14"
Oct 30 23:49:30.771786 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.771629    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11079-163a14"]
Oct 30 23:49:30.797513 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:30.795146    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-914b2c\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="01b3bbfc-856e-446c-a099-8b36d99e0525" pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:30.838404 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:30.832965    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-86556d\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="269dc58d-0b23-4f44-9f73-3910c85a3cfb" pod="opendatahub/test-curl-11064-86556d"
Oct 30 23:49:30.838404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.838287    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7c5b\" (UniqueName: \"kubernetes.io/projected/1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54-kube-api-access-m7c5b\") pod \"test-curl-11079-163a14\" (UID: \"1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54\") " pod="opendatahub/test-curl-11079-163a14"
Oct 30 23:49:30.940092 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.940059    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7c5b\" (UniqueName: \"kubernetes.io/projected/1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54-kube-api-access-m7c5b\") pod \"test-curl-11079-163a14\" (UID: \"1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54\") " pod="opendatahub/test-curl-11079-163a14"
Oct 30 23:49:30.952647 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:30.952612    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7c5b\" (UniqueName: \"kubernetes.io/projected/1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54-kube-api-access-m7c5b\") pod \"test-curl-11079-163a14\" (UID: \"1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54\") " pod="opendatahub/test-curl-11079-163a14"
Oct 30 23:49:31.134907 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:31.134866    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-163a14"
Oct 30 23:49:31.312358 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:31.298648    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-86556d\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="269dc58d-0b23-4f44-9f73-3910c85a3cfb" pod="opendatahub/test-curl-11064-86556d"
Oct 30 23:49:31.312358 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:31.301206    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11064-914b2c\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="01b3bbfc-856e-446c-a099-8b36d99e0525" pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:31.312683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:31.312559    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269dc58d-0b23-4f44-9f73-3910c85a3cfb" path="/var/lib/kubelet/pods/269dc58d-0b23-4f44-9f73-3910c85a3cfb/volumes"
Oct 30 23:49:31.568461 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:49:31.565474    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c9cdcc5_7da6_4089_8bae_1bb79cfbbe54.slice/crio-c02409c4af3958edcc4dde657c2d9ea0952338916bb16dddb21f8227eba42425 WatchSource:0}: Error finding container c02409c4af3958edcc4dde657c2d9ea0952338916bb16dddb21f8227eba42425: Status 404 returned error can't find the container with id c02409c4af3958edcc4dde657c2d9ea0952338916bb16dddb21f8227eba42425
Oct 30 23:49:31.568461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:31.566780    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11079-163a14"]
Oct 30 23:49:32.524672 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:32.524632    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11079-163a14" event={"ID":"1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54","Type":"ContainerStarted","Data":"fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473"}
Oct 30 23:49:32.525217 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:32.524685    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11079-163a14" event={"ID":"1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54","Type":"ContainerStarted","Data":"c02409c4af3958edcc4dde657c2d9ea0952338916bb16dddb21f8227eba42425"}
Oct 30 23:49:32.551072 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:32.551009    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11079-163a14" podStartSLOduration=2.550988686 podStartE2EDuration="2.550988686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:32.546744769 +0000 UTC m=+1831.473382368" watchObservedRunningTime="2026-10-30 23:49:32.550988686 +0000 UTC m=+1831.477626268"
Oct 30 23:49:32.838422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:32.835832    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:32.913618 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:32.913589    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b3bbfc-856e-446c-a099-8b36d99e0525-kube-api-access-h8fwl\" (UniqueName: \"kubernetes.io/projected/01b3bbfc-856e-446c-a099-8b36d99e0525-kube-api-access-h8fwl\") pod \"01b3bbfc-856e-446c-a099-8b36d99e0525\" (UID: \"01b3bbfc-856e-446c-a099-8b36d99e0525\") "
Oct 30 23:49:32.916449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:32.916411    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b3bbfc-856e-446c-a099-8b36d99e0525-kube-api-access-h8fwl" pod "01b3bbfc-856e-446c-a099-8b36d99e0525" (UID: "01b3bbfc-856e-446c-a099-8b36d99e0525"). InnerVolumeSpecName "kube-api-access-h8fwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:33.016270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.016236    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8fwl\" (UniqueName: \"kubernetes.io/projected/01b3bbfc-856e-446c-a099-8b36d99e0525-kube-api-access-h8fwl\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:33.211217 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.211131    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b3bbfc-856e-446c-a099-8b36d99e0525" path="/var/lib/kubelet/pods/01b3bbfc-856e-446c-a099-8b36d99e0525/volumes"
Oct 30 23:49:33.372178 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.372123    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11079-163a14"]
Oct 30 23:49:33.378619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.378588    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11079-163a14"]
Oct 30 23:49:33.534044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.534004    4555 generic.go:356] "Generic (PLEG): container finished" podID="01b3bbfc-856e-446c-a099-8b36d99e0525" containerID="946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b" exitCode=137
Oct 30 23:49:33.534551 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.534052    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11064-914b2c"
Oct 30 23:49:33.534551 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.534111    4555 scope.go:122] "RemoveContainer" containerID="946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b"
Oct 30 23:49:33.545939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.545916    4555 scope.go:122] "RemoveContainer" containerID="946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b"
Oct 30 23:49:33.546236 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:33.546210    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b\": container with ID starting with 946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b not found: ID does not exist" containerID="946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b"
Oct 30 23:49:33.546287 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.546247    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b"} err="failed to get container status \"946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b\": rpc error: code = NotFound desc = could not find container \"946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b\": container with ID starting with 946bffa87ec7b89cf35b2298f74acce56bbc8dfdc497ff45618562331d1b788b not found: ID does not exist"
Oct 30 23:49:33.623060 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.623026    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"]
Oct 30 23:49:33.623858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.623833    4555 cpu_manager.go:414] "containerMap: removing container" podUID="1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54" containerName="test-curl-11079-163a14"
Oct 30 23:49:33.623858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.623855    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54" containerName="test-curl-11079-163a14"
Oct 30 23:49:33.624035 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.623964    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54" containerName="test-curl-11079-163a14"
Oct 30 23:49:33.628979 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.628960    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:33.634355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.634059    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:49:33.634355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.634115    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:49:33.637287 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.637263    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"]
Oct 30 23:49:33.725123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.725089    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-s6rqf\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:33.725307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.725219    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8l5\" (UniqueName: \"kubernetes.io/projected/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-kube-api-access-vj8l5\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-s6rqf\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:33.725307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.725275    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-s6rqf\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:33.826524 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.826432    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-s6rqf\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:33.826688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.826562    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8l5\" (UniqueName: \"kubernetes.io/projected/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-kube-api-access-vj8l5\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-s6rqf\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:33.826688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.826622    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-s6rqf\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:33.827281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.827262    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-s6rqf\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:33.829071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.829049    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-s6rqf\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:33.838962 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.838939    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8l5\" (UniqueName: \"kubernetes.io/projected/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-kube-api-access-vj8l5\") pod \"maas-api-revoke-keys-e2e-mt-6a162557-s6rqf\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:33.947715 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:33.947679    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:34.335150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:34.335117    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"]
Oct 30 23:49:34.335473 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:49:34.335451    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00dc2d25_17d4_4efa_aa39_f4e83d9f0564.slice/crio-074004fe2f9fe543653017933a8df963d1c19d6ea0b387b59da65b2d97a9a5b6 WatchSource:0}: Error finding container 074004fe2f9fe543653017933a8df963d1c19d6ea0b387b59da65b2d97a9a5b6: Status 404 returned error can't find the container with id 074004fe2f9fe543653017933a8df963d1c19d6ea0b387b59da65b2d97a9a5b6
Oct 30 23:49:34.544799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:34.544760    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf" event={"ID":"00dc2d25-17d4-4efa-aa39-f4e83d9f0564","Type":"ContainerStarted","Data":"074004fe2f9fe543653017933a8df963d1c19d6ea0b387b59da65b2d97a9a5b6"}
Oct 30 23:49:34.545260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:34.544833    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11079-163a14" podUID="1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54" containerName="test-curl-11079-163a14" containerID="cri-o://fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473" gracePeriod=2
Oct 30 23:49:34.548002 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:34.547969    4555 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11079-163a14\" is forbidden: User \"system:node:ip-10-0-134-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-east-2.compute.internal' and this object" podUID="1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54" pod="opendatahub/test-curl-11079-163a14"
Oct 30 23:49:35.554596 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:35.554562    4555 generic.go:356] "Generic (PLEG): container finished" podID="00dc2d25-17d4-4efa-aa39-f4e83d9f0564" containerID="9cc1708765dea9f34ec30d04a6d53ec041e9c8f77a0d42af5a1397d42fa49985" exitCode=0
Oct 30 23:49:35.554996 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:35.554648    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf" event={"ID":"00dc2d25-17d4-4efa-aa39-f4e83d9f0564","Type":"ContainerDied","Data":"9cc1708765dea9f34ec30d04a6d53ec041e9c8f77a0d42af5a1397d42fa49985"}
Oct 30 23:49:36.196824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:36.196789    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:49:36.583429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:36.582557    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:36.644224 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:36.643877    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:36.968319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:36.968280    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:37.034653 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.034623    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-163a14"
Oct 30 23:49:37.081194 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.081156    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-xks-ca\") pod \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") "
Oct 30 23:49:37.081413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.081256    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-kube-api-access-vj8l5\" (UniqueName: \"kubernetes.io/projected/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-kube-api-access-vj8l5\") pod \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") "
Oct 30 23:49:37.081498 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.081480    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-openshift-ca\") pod \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\" (UID: \"00dc2d25-17d4-4efa-aa39-f4e83d9f0564\") "
Oct 30 23:49:37.082284 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.081985    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-openshift-ca" pod "00dc2d25-17d4-4efa-aa39-f4e83d9f0564" (UID: "00dc2d25-17d4-4efa-aa39-f4e83d9f0564"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:49:37.082284 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.082224    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:37.084086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.084051    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-xks-ca" pod "00dc2d25-17d4-4efa-aa39-f4e83d9f0564" (UID: "00dc2d25-17d4-4efa-aa39-f4e83d9f0564"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:49:37.084635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.084606    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-kube-api-access-vj8l5" pod "00dc2d25-17d4-4efa-aa39-f4e83d9f0564" (UID: "00dc2d25-17d4-4efa-aa39-f4e83d9f0564"). InnerVolumeSpecName "kube-api-access-vj8l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:37.191401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.183919    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54-kube-api-access-m7c5b\" (UniqueName: \"kubernetes.io/projected/1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54-kube-api-access-m7c5b\") pod \"1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54\" (UID: \"1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54\") "
Oct 30 23:49:37.210214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.210174    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54-kube-api-access-m7c5b" pod "1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54" (UID: "1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54"). InnerVolumeSpecName "kube-api-access-m7c5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:37.213771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.213737    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:37.213971 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.213956    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7c5b\" (UniqueName: \"kubernetes.io/projected/1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54-kube-api-access-m7c5b\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:37.214085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.214072    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8l5\" (UniqueName: \"kubernetes.io/projected/00dc2d25-17d4-4efa-aa39-f4e83d9f0564-kube-api-access-vj8l5\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:37.261621 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.261584    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54" path="/var/lib/kubelet/pods/1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54/volumes"
Oct 30 23:49:37.601053 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.600956    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:49:37.630488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.630450    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"
Oct 30 23:49:37.630664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.630440    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf" event={"ID":"00dc2d25-17d4-4efa-aa39-f4e83d9f0564","Type":"ContainerDied","Data":"074004fe2f9fe543653017933a8df963d1c19d6ea0b387b59da65b2d97a9a5b6"}
Oct 30 23:49:37.630664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.630589    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="074004fe2f9fe543653017933a8df963d1c19d6ea0b387b59da65b2d97a9a5b6"
Oct 30 23:49:37.632181 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.632155    4555 generic.go:356] "Generic (PLEG): container finished" podID="1c9cdcc5-7da6-4089-8bae-1bb79cfbbe54" containerID="fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473" exitCode=137
Oct 30 23:49:37.632348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.632202    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11079-163a14"
Oct 30 23:49:37.632348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.632289    4555 scope.go:122] "RemoveContainer" containerID="fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473"
Oct 30 23:49:37.644429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.644410    4555 scope.go:122] "RemoveContainer" containerID="fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473"
Oct 30 23:49:37.644746 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:37.644721    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473\": container with ID starting with fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473 not found: ID does not exist" containerID="fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473"
Oct 30 23:49:37.644821 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:37.644758    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473"} err="failed to get container status \"fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473\": rpc error: code = NotFound desc = could not find container \"fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473\": container with ID starting with fbb1ea52acc1d0081537d0a2e81ec7c6fd69cf5e4822716dc68b1d7c59bc5473 not found: ID does not exist"
Oct 30 23:49:39.357138 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.357107    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"]
Oct 30 23:49:39.358775 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.358751    4555 cpu_manager.go:414] "containerMap: removing container" podUID="00dc2d25-17d4-4efa-aa39-f4e83d9f0564" containerName="revoke-keys"
Oct 30 23:49:39.358932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.358919    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00dc2d25-17d4-4efa-aa39-f4e83d9f0564" containerName="revoke-keys"
Oct 30 23:49:39.359254 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.359237    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="00dc2d25-17d4-4efa-aa39-f4e83d9f0564" containerName="revoke-keys"
Oct 30 23:49:39.375431 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.369905    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.407641 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.390335    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1bc837\"/\"kube-root-ca.crt\""
Oct 30 23:49:39.408653 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.408628    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1bc837\"/\"openshift-service-ca.crt\""
Oct 30 23:49:39.408928 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.408910    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1bc837\"/\"test-model-1bc837-kserve-self-signed-certs\""
Oct 30 23:49:39.412591 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.409905    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1bc837\"/\"default-dockercfg-9pn8j\""
Oct 30 23:49:39.412591 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.410927    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"]
Oct 30 23:49:39.434839 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.434810    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcs2d\" (UniqueName: \"kubernetes.io/projected/1b014885-3f0e-43d6-bf70-de300c1ae2db-kube-api-access-gcs2d\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.435062 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.435044    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-tmp-dir\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.435251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.435232    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-home\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.435655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.435636    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-dshm\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.435826 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.435807    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b014885-3f0e-43d6-bf70-de300c1ae2db-tls-certs\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.435962 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.435947    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-model-cache\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.547997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.547940    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcs2d\" (UniqueName: \"kubernetes.io/projected/1b014885-3f0e-43d6-bf70-de300c1ae2db-kube-api-access-gcs2d\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.548325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.548290    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-tmp-dir\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.551999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.551811    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-home\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.552267 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.552246    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-dshm\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.552463 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.552447    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b014885-3f0e-43d6-bf70-de300c1ae2db-tls-certs\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.552779 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.552758    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-model-cache\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.554250 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.554215    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-home\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.555097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.554692    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-model-cache\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.555939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.555901    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-tmp-dir\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.577108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.577075    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-dshm\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.657450 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.656881    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcs2d\" (UniqueName: \"kubernetes.io/projected/1b014885-3f0e-43d6-bf70-de300c1ae2db-kube-api-access-gcs2d\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.660989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.660949    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b014885-3f0e-43d6-bf70-de300c1ae2db-tls-certs\") pod \"test-model-1bc837-kserve-744b8cb76c-kq57g\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") " pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:39.743587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:39.743547    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:40.585490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:40.584000    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"]
Oct 30 23:49:40.831342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:40.831307    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" event={"ID":"1b014885-3f0e-43d6-bf70-de300c1ae2db","Type":"ContainerStarted","Data":"655b49fffa2587b570383e3c16d87fc30739ea2d3804a8973c74c1b3fde3e47a"}
Oct 30 23:49:41.846305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:41.846203    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" event={"ID":"1b014885-3f0e-43d6-bf70-de300c1ae2db","Type":"ContainerStarted","Data":"ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee"}
Oct 30 23:49:41.861001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:41.860970    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"]
Oct 30 23:49:41.861468 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:41.861437    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy" containerID="cri-o://2967f7f09edd2710845679214633df20b74a56889fb9ec685882519e826102b7" gracePeriod=30
Oct 30 23:49:41.880644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:41.880581    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" podStartSLOduration=2.880562317 podStartE2EDuration="2.880562317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:41.872019614 +0000 UTC m=+1840.798657196" watchObservedRunningTime="2026-10-30 23:49:41.880562317 +0000 UTC m=+1840.807199899"
Oct 30 23:49:42.744807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:42.744771    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:47.091654 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.091305    4555 generic.go:356] "Generic (PLEG): container finished" podID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerID="2967f7f09edd2710845679214633df20b74a56889fb9ec685882519e826102b7" exitCode=0
Oct 30 23:49:47.091654 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.091581    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" event={"ID":"44f7cf65-6501-4af8-81af-9e2c0c123b57","Type":"ContainerDied","Data":"2967f7f09edd2710845679214633df20b74a56889fb9ec685882519e826102b7"}
Oct 30 23:49:47.411987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.411948    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:49:47.522571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.522530    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-token\" (UniqueName: \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-token\") pod \"44f7cf65-6501-4af8-81af-9e2c0c123b57\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") "
Oct 30 23:49:47.522828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.522815    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-certs\") pod \"44f7cf65-6501-4af8-81af-9e2c0c123b57\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") "
Oct 30 23:49:47.522934 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.522922    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-podinfo\") pod \"44f7cf65-6501-4af8-81af-9e2c0c123b57\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") "
Oct 30 23:49:47.523099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.523083    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-credential-socket\") pod \"44f7cf65-6501-4af8-81af-9e2c0c123b57\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") "
Oct 30 23:49:47.523410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.523155    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-certs" pod "44f7cf65-6501-4af8-81af-9e2c0c123b57" (UID: "44f7cf65-6501-4af8-81af-9e2c0c123b57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:47.523410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.523346    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-credential-socket" pod "44f7cf65-6501-4af8-81af-9e2c0c123b57" (UID: "44f7cf65-6501-4af8-81af-9e2c0c123b57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:47.523666 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.523646    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-kube-api-access-l6vwt\" (UniqueName: \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-kube-api-access-l6vwt\") pod \"44f7cf65-6501-4af8-81af-9e2c0c123b57\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") "
Oct 30 23:49:47.523810 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.523794    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-data\") pod \"44f7cf65-6501-4af8-81af-9e2c0c123b57\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") "
Oct 30 23:49:47.524026 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.523990    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-envoy\") pod \"44f7cf65-6501-4af8-81af-9e2c0c123b57\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") "
Oct 30 23:49:47.524178 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.524161    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f7cf65-6501-4af8-81af-9e2c0c123b57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f7cf65-6501-4af8-81af-9e2c0c123b57-istiod-ca-cert\") pod \"44f7cf65-6501-4af8-81af-9e2c0c123b57\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") "
Oct 30 23:49:47.524446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.524426    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-socket\") pod \"44f7cf65-6501-4af8-81af-9e2c0c123b57\" (UID: \"44f7cf65-6501-4af8-81af-9e2c0c123b57\") "
Oct 30 23:49:47.525960 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.525341    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:47.526097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.526082    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:47.526472 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.526452    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-socket" pod "44f7cf65-6501-4af8-81af-9e2c0c123b57" (UID: "44f7cf65-6501-4af8-81af-9e2c0c123b57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:47.526830 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.526817    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f7cf65-6501-4af8-81af-9e2c0c123b57-istiod-ca-cert" pod "44f7cf65-6501-4af8-81af-9e2c0c123b57" (UID: "44f7cf65-6501-4af8-81af-9e2c0c123b57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:49:47.527063 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.527052    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-data" pod "44f7cf65-6501-4af8-81af-9e2c0c123b57" (UID: "44f7cf65-6501-4af8-81af-9e2c0c123b57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:47.533488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.533460    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-podinfo" pod "44f7cf65-6501-4af8-81af-9e2c0c123b57" (UID: "44f7cf65-6501-4af8-81af-9e2c0c123b57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:49:47.533614 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.533592    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-token" pod "44f7cf65-6501-4af8-81af-9e2c0c123b57" (UID: "44f7cf65-6501-4af8-81af-9e2c0c123b57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:47.533680 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.533661    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-envoy" pod "44f7cf65-6501-4af8-81af-9e2c0c123b57" (UID: "44f7cf65-6501-4af8-81af-9e2c0c123b57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:47.533974 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.533933    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-kube-api-access-l6vwt" pod "44f7cf65-6501-4af8-81af-9e2c0c123b57" (UID: "44f7cf65-6501-4af8-81af-9e2c0c123b57"). InnerVolumeSpecName "kube-api-access-l6vwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:47.558634 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.558603    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"]
Oct 30 23:49:47.566255 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.566221    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a162557-s6rqf"]
Oct 30 23:49:47.627819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.627725    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6vwt\" (UniqueName: \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-kube-api-access-l6vwt\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:47.627819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.627772    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:47.627819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.627787    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:47.627819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.627800    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f7cf65-6501-4af8-81af-9e2c0c123b57-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:47.627819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.627815    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f7cf65-6501-4af8-81af-9e2c0c123b57-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:47.628093 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.627828    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:47.628093 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:47.627842    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f7cf65-6501-4af8-81af-9e2c0c123b57-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:48.111228 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:48.111187    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8" event={"ID":"44f7cf65-6501-4af8-81af-9e2c0c123b57","Type":"ContainerDied","Data":"48a08d38b0c5fc236bb2c565341d15c56ba1a19d75934b3867ed830b163c7fe3"}
Oct 30 23:49:48.111778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:48.111270    4555 scope.go:122] "RemoveContainer" containerID="2967f7f09edd2710845679214633df20b74a56889fb9ec685882519e826102b7"
Oct 30 23:49:48.111778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:48.111425    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"
Oct 30 23:49:48.161019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:48.160943    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"]
Oct 30 23:49:48.168624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:48.168585    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-whbc8"]
Oct 30 23:49:49.212394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:49.211587    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00dc2d25-17d4-4efa-aa39-f4e83d9f0564" path="/var/lib/kubelet/pods/00dc2d25-17d4-4efa-aa39-f4e83d9f0564/volumes"
Oct 30 23:49:49.212394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:49.212200    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" path="/var/lib/kubelet/pods/44f7cf65-6501-4af8-81af-9e2c0c123b57/volumes"
Oct 30 23:49:49.746664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:49.746625    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:49.759952 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:49.759925    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:50.207799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:50.207764    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:49:51.300636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:51.300321    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": read tcp 10.133.0.2:35934->10.133.0.132:15021: read: connection reset by peer" start-of-body=
Oct 30 23:49:51.300636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:51.300413    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": read tcp 10.133.0.2:35934->10.133.0.132:15021: read: connection reset by peer"
Oct 30 23:49:52.298279 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:52.298242    4555 generic.go:356] "Generic (PLEG): container finished" podID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerID="03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01" exitCode=0
Oct 30 23:49:52.298501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:52.298466    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" event={"ID":"76874aeb-95f4-483e-acca-cb1ba0b9ecfc","Type":"ContainerDied","Data":"03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01"}
Oct 30 23:49:52.300202 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:52.300180    4555 scope.go:122] "RemoveContainer" containerID="03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01"
Oct 30 23:49:52.319914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:52.314941    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:49:52.319914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:52.315024    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:49:52.319914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:52.315058    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:49:53.016135 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:53.016103    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"]
Oct 30 23:49:53.311997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:53.311954    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" event={"ID":"76874aeb-95f4-483e-acca-cb1ba0b9ecfc","Type":"ContainerStarted","Data":"9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a"}
Oct 30 23:49:53.604735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:53.604638    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:49:53.605511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:53.605329    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 23:49:53.605708 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:53.605685    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 23:49:54.602708 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:54.602672    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 23:49:54.602936 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:54.602733    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 23:49:55.602707 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:55.602645    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 23:49:55.603241 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:55.602710    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 23:49:56.605889 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:56.605852    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 23:49:56.606535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:56.606506    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 23:49:57.450485 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:49:57.450449    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:49:57.604807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:57.604771    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 23:49:57.605112 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:57.605082    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 23:49:58.602845 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:58.602803    4555 patch_prober.go:29] interesting pod/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 23:49:58.603261 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:58.602884    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 23:49:59.607767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:59.607735    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:49:59.608259 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:59.608244    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:49:59.609043 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:49:59.609021    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 30 23:50:02.837437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:02.837406    4555 scope.go:122] "RemoveContainer" containerID="73440c8e423ef469a165da6ed4f3128732b3086e20b2591c20f4ec210b135279"
Oct 30 23:50:02.849973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:02.849946    4555 scope.go:122] "RemoveContainer" containerID="92987329f85ecf15d9587eed3c23aae2ee0bf6223f6b6e5364612c11dbd6a169"
Oct 30 23:50:02.861842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:02.861819    4555 scope.go:122] "RemoveContainer" containerID="1bbbaa93df5f304d4353779b02aeba208f816147f6038b069025749ba2d81ca5"
Oct 30 23:50:02.872331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:02.872311    4555 scope.go:122] "RemoveContainer" containerID="29bd34534c38f6233f180063516a6048e22d966cb2288ecad1f7a445751ba728"
Oct 30 23:50:04.074429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.074394    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"]
Oct 30 23:50:04.075445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.075422    4555 cpu_manager.go:414] "containerMap: removing container" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy"
Oct 30 23:50:04.075675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.075659    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy"
Oct 30 23:50:04.077457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.077432    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f7cf65-6501-4af8-81af-9e2c0c123b57" containerName="istio-proxy"
Oct 30 23:50:04.092404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.092334    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"]
Oct 30 23:50:04.092654 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.092635    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.098096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.098074    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-23f75e-serving-cert\""
Oct 30 23:50:04.133452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.133415    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-maas-api-tls\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.133651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.133587    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5smld\" (UniqueName: \"kubernetes.io/projected/78ae695d-9222-4a0a-900b-441c8486efbc-kube-api-access-5smld\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.133651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.133636    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ae695d-9222-4a0a-900b-441c8486efbc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.133743 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.133723    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-metrics-certs\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.239397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.238680    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-maas-api-tls\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.265478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.254018    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5smld\" (UniqueName: \"kubernetes.io/projected/78ae695d-9222-4a0a-900b-441c8486efbc-kube-api-access-5smld\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.265478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.254093    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ae695d-9222-4a0a-900b-441c8486efbc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.265478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.254129    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-metrics-certs\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.265478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.258088    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-maas-api-tls\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.265478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.258203    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ae695d-9222-4a0a-900b-441c8486efbc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.286392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.285073    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-metrics-certs\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.302954 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.300548    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5smld\" (UniqueName: \"kubernetes.io/projected/78ae695d-9222-4a0a-900b-441c8486efbc-kube-api-access-5smld\") pod \"maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.414989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.414894    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:04.856396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:04.856334    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"]
Oct 30 23:50:04.873402 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:50:04.864020    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ae695d_9222_4a0a_900b_441c8486efbc.slice/crio-2e1cc64a415b223114f2c5a1dead8331976a83480401966bd44254d711d21e92 WatchSource:0}: Error finding container 2e1cc64a415b223114f2c5a1dead8331976a83480401966bd44254d711d21e92: Status 404 returned error can't find the container with id 2e1cc64a415b223114f2c5a1dead8331976a83480401966bd44254d711d21e92
Oct 30 23:50:05.567631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:05.567590    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh" event={"ID":"78ae695d-9222-4a0a-900b-441c8486efbc","Type":"ContainerStarted","Data":"3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46"}
Oct 30 23:50:05.568101 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:05.567659    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh" event={"ID":"78ae695d-9222-4a0a-900b-441c8486efbc","Type":"ContainerStarted","Data":"2e1cc64a415b223114f2c5a1dead8331976a83480401966bd44254d711d21e92"}
Oct 30 23:50:05.568101 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:05.567688    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:05.589957 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:05.589897    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh" podStartSLOduration=1.125542148 podStartE2EDuration="1.589880539s" totalImagesPullingTime="464.338391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:05.587419141 +0000 UTC m=+1864.514056720" watchObservedRunningTime="2026-10-30 23:50:05.589880539 +0000 UTC m=+1864.516518119"
Oct 30 23:50:08.042042 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.042004    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd" podUID="761237ce-49dc-4f27-9a7e-cd5359d77257" containerName="main" containerID="cri-o://29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15" gracePeriod=45
Oct 30 23:50:08.344807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.344776    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:50:08.384416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.384350    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-dshm\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-dshm\") pod \"761237ce-49dc-4f27-9a7e-cd5359d77257\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") "
Oct 30 23:50:08.384638 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.384619    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-home\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-home\") pod \"761237ce-49dc-4f27-9a7e-cd5359d77257\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") "
Oct 30 23:50:08.384723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.384663    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761237ce-49dc-4f27-9a7e-cd5359d77257-kube-api-access-v6b4n\" (UniqueName: \"kubernetes.io/projected/761237ce-49dc-4f27-9a7e-cd5359d77257-kube-api-access-v6b4n\") pod \"761237ce-49dc-4f27-9a7e-cd5359d77257\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") "
Oct 30 23:50:08.384723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.384708    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761237ce-49dc-4f27-9a7e-cd5359d77257-tls-certs\" (UniqueName: \"kubernetes.io/secret/761237ce-49dc-4f27-9a7e-cd5359d77257-tls-certs\") pod \"761237ce-49dc-4f27-9a7e-cd5359d77257\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") "
Oct 30 23:50:08.384899 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.384803    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-tmp-dir\") pod \"761237ce-49dc-4f27-9a7e-cd5359d77257\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") "
Oct 30 23:50:08.384899 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.384872    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-model-cache\") pod \"761237ce-49dc-4f27-9a7e-cd5359d77257\" (UID: \"761237ce-49dc-4f27-9a7e-cd5359d77257\") "
Oct 30 23:50:08.385491 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.385456    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-home" pod "761237ce-49dc-4f27-9a7e-cd5359d77257" (UID: "761237ce-49dc-4f27-9a7e-cd5359d77257"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:50:08.385945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.385659    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-tmp-dir" pod "761237ce-49dc-4f27-9a7e-cd5359d77257" (UID: "761237ce-49dc-4f27-9a7e-cd5359d77257"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:50:08.385945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.385759    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-model-cache" pod "761237ce-49dc-4f27-9a7e-cd5359d77257" (UID: "761237ce-49dc-4f27-9a7e-cd5359d77257"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:50:08.385945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.385784    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:08.388693 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.388662    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761237ce-49dc-4f27-9a7e-cd5359d77257-kube-api-access-v6b4n" pod "761237ce-49dc-4f27-9a7e-cd5359d77257" (UID: "761237ce-49dc-4f27-9a7e-cd5359d77257"). InnerVolumeSpecName "kube-api-access-v6b4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:50:08.389905 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.389849    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-dshm" pod "761237ce-49dc-4f27-9a7e-cd5359d77257" (UID: "761237ce-49dc-4f27-9a7e-cd5359d77257"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:50:08.390459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.390431    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761237ce-49dc-4f27-9a7e-cd5359d77257-tls-certs" pod "761237ce-49dc-4f27-9a7e-cd5359d77257" (UID: "761237ce-49dc-4f27-9a7e-cd5359d77257"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:50:08.490578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.490530    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6b4n\" (UniqueName: \"kubernetes.io/projected/761237ce-49dc-4f27-9a7e-cd5359d77257-kube-api-access-v6b4n\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:08.490578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.490579    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761237ce-49dc-4f27-9a7e-cd5359d77257-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:08.490843 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.490594    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:08.490843 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.490608    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:08.490843 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.490621    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761237ce-49dc-4f27-9a7e-cd5359d77257-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:08.598438 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.598299    4555 generic.go:356] "Generic (PLEG): container finished" podID="761237ce-49dc-4f27-9a7e-cd5359d77257" containerID="29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15" exitCode=0
Oct 30 23:50:08.598625 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.598478    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"
Oct 30 23:50:08.598625 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.598487    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd" event={"ID":"761237ce-49dc-4f27-9a7e-cd5359d77257","Type":"ContainerDied","Data":"29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15"}
Oct 30 23:50:08.598625 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.598553    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd" event={"ID":"761237ce-49dc-4f27-9a7e-cd5359d77257","Type":"ContainerDied","Data":"95d4eb4b16bdcf32246de8471092ca6f0294ad094d2fe0604a8ca4be15b97326"}
Oct 30 23:50:08.598625 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.598571    4555 scope.go:122] "RemoveContainer" containerID="29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15"
Oct 30 23:50:08.612232 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.612201    4555 scope.go:122] "RemoveContainer" containerID="29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15"
Oct 30 23:50:08.612624 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:50:08.612601    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15\": container with ID starting with 29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15 not found: ID does not exist" containerID="29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15"
Oct 30 23:50:08.612728 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.612634    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15"} err="failed to get container status \"29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15\": rpc error: code = NotFound desc = could not find container \"29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15\": container with ID starting with 29fb5c406bc4390610d01a329161f1454ae74c069413e43dcd6fe51e72ec3e15 not found: ID does not exist"
Oct 30 23:50:08.630121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.630092    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"]
Oct 30 23:50:08.634798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:08.634774    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd"]
Oct 30 23:50:09.212187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.212156    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761237ce-49dc-4f27-9a7e-cd5359d77257" path="/var/lib/kubelet/pods/761237ce-49dc-4f27-9a7e-cd5359d77257/volumes"
Oct 30 23:50:09.252071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.252033    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"]
Oct 30 23:50:09.252909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.252878    4555 cpu_manager.go:414] "containerMap: removing container" podUID="761237ce-49dc-4f27-9a7e-cd5359d77257" containerName="main"
Oct 30 23:50:09.252909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.252900    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761237ce-49dc-4f27-9a7e-cd5359d77257" containerName="main"
Oct 30 23:50:09.253062 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.253035    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="761237ce-49dc-4f27-9a7e-cd5359d77257" containerName="main"
Oct 30 23:50:09.258116 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.258090    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.267421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.265977    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-6a162557/e2e-lifecycle-model-6a162557-kserve-56f4fd5489-r7xsd" podUID="761237ce-49dc-4f27-9a7e-cd5359d77257" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded"
Oct 30 23:50:09.267421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.266678    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-82d449-serving-cert\""
Oct 30 23:50:09.271876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.271624    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"]
Oct 30 23:50:09.314474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.314438    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.314671 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.314562    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-metrics-certs\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.314671 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.314647    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw7w8\" (UniqueName: \"kubernetes.io/projected/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-kube-api-access-zw7w8\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.314830 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.314812    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-maas-api-tls\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.415745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.415708    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-metrics-certs\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.415925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.415812    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw7w8\" (UniqueName: \"kubernetes.io/projected/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-kube-api-access-zw7w8\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.416035 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.416012    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-maas-api-tls\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.416221 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.416198    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.416990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.416963    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.418644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.418613    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-metrics-certs\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.419079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.419058    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-maas-api-tls\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.445878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.445811    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw7w8\" (UniqueName: \"kubernetes.io/projected/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-kube-api-access-zw7w8\") pod \"maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.587095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.587064    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:09.895416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:09.889027    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"]
Oct 30 23:50:09.895416 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:50:09.890094    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebbcb45_d81e_43a4_b925_bd65f76e3c9b.slice/crio-1605c0e3abed68384a47022167ee5a428a4cefe3d2cc045a926639a29a1eb22f WatchSource:0}: Error finding container 1605c0e3abed68384a47022167ee5a428a4cefe3d2cc045a926639a29a1eb22f: Status 404 returned error can't find the container with id 1605c0e3abed68384a47022167ee5a428a4cefe3d2cc045a926639a29a1eb22f
Oct 30 23:50:10.618657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:10.618564    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk" event={"ID":"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b","Type":"ContainerStarted","Data":"bb760cf055b0d990106410be8175ce04043cbe2b3734459d2dd1b4f65eff09cd"}
Oct 30 23:50:10.618657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:10.618632    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:10.618657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:10.618653    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk" event={"ID":"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b","Type":"ContainerStarted","Data":"1605c0e3abed68384a47022167ee5a428a4cefe3d2cc045a926639a29a1eb22f"}
Oct 30 23:50:10.642829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:10.642757    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk" podStartSLOduration=1.213547543 podStartE2EDuration="1.642738528s" totalImagesPullingTime="429.190985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:10.640829628 +0000 UTC m=+1869.567467209" watchObservedRunningTime="2026-10-30 23:50:10.642738528 +0000 UTC m=+1869.569376109"
Oct 30 23:50:11.584394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:11.584071    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:50:15.850017 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:15.849975    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"]
Oct 30 23:50:15.854807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:15.854786    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:50:15.861388 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:15.861350    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f0817c-dockercfg-9njvf\""
Oct 30 23:50:15.889052 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:15.888976    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"]
Oct 30 23:50:15.918549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:15.918517    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f0817c\""
Oct 30 23:50:16.022393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:16.020164    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhwdc\" (UniqueName: \"kubernetes.io/projected/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-kube-api-access-bhwdc\") pod \"payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt\" (UID: \"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:50:16.022393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:16.020242    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt\" (UID: \"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:50:16.143521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:16.142508    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhwdc\" (UniqueName: \"kubernetes.io/projected/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-kube-api-access-bhwdc\") pod \"payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt\" (UID: \"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:50:16.143521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:16.142589    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt\" (UID: \"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:50:16.149397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:16.143808    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt\" (UID: \"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:50:16.200730 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:16.200690    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhwdc\" (UniqueName: \"kubernetes.io/projected/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-kube-api-access-bhwdc\") pod \"payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt\" (UID: \"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:50:16.475627 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:16.475535    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:50:16.907159 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:16.903747    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:50:16.983553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:16.983492    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"]
Oct 30 23:50:18.019039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:18.018997    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt" event={"ID":"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec","Type":"ContainerStarted","Data":"58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38"}
Oct 30 23:50:18.019545 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:18.019057    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt" event={"ID":"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec","Type":"ContainerStarted","Data":"cabd5552a23cfc9ce184ad1435b107135d1392bb83291153d88859d491d9966d"}
Oct 30 23:50:18.019545 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:18.019098    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:50:18.040043 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:18.039990    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt" podStartSLOduration=3.03997546 podStartE2EDuration="3.03997546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:18.03712184 +0000 UTC m=+1876.963759420" watchObservedRunningTime="2026-10-30 23:50:18.03997546 +0000 UTC m=+1876.966613041"
Oct 30 23:50:25.193307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:25.193270    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:50:27.543101 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:50:27.543062    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:50:27.585003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.584936    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"]
Oct 30 23:50:27.603424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.603398    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.611838 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.611811    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"]
Oct 30 23:50:27.619131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.617567    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f026c1\"/\"default-dockercfg-pkssz\""
Oct 30 23:50:27.619131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.617670    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f026c1\"/\"kube-root-ca.crt\""
Oct 30 23:50:27.619131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.617759    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f026c1\"/\"test-model-f026c1-kserve-self-signed-certs\""
Oct 30 23:50:27.671394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.671341    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-model-cache\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.671598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.671553    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqq6\" (UniqueName: \"kubernetes.io/projected/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-kube-api-access-8lqq6\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.671668 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.671599    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tls-certs\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.671728 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.671691    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-dshm\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.671728 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.671717    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-home\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.671854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.671833    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tmp-dir\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.715431 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.715393    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f026c1\"/\"openshift-service-ca.crt\""
Oct 30 23:50:27.773387 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.773337    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqq6\" (UniqueName: \"kubernetes.io/projected/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-kube-api-access-8lqq6\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.773577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.773398    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tls-certs\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.773577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.773458    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-dshm\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.773704 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.773574    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-home\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.773761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.773719    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tmp-dir\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.774063 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.773849    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-model-cache\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.774193 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.774100    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-home\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.774193 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.774169    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tmp-dir\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.774329 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.774306    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-model-cache\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.776542 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.776515    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-dshm\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.776773 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.776755    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tls-certs\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.786060 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.786039    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqq6\" (UniqueName: \"kubernetes.io/projected/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-kube-api-access-8lqq6\") pod \"test-model-f026c1-kserve-6675c47c8c-x9zx4\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") " pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:27.926105 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:27.924751    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:28.481111 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:50:28.481067    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61c076ae_c401_4b0a_b3e6_d358bf9dd00c.slice/crio-b92301ab7b49e7234452aba7f68ada5a94e3a9645db3504ac094b551fe5de1d2 WatchSource:0}: Error finding container b92301ab7b49e7234452aba7f68ada5a94e3a9645db3504ac094b551fe5de1d2: Status 404 returned error can't find the container with id b92301ab7b49e7234452aba7f68ada5a94e3a9645db3504ac094b551fe5de1d2
Oct 30 23:50:28.481526 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:28.481502    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"]
Oct 30 23:50:29.030790 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:29.030756    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:50:29.229754 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:29.229715    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4" event={"ID":"61c076ae-c401-4b0a-b3e6-d358bf9dd00c","Type":"ContainerStarted","Data":"369249b4a410e7b8f4215bf84ab464861f0f3d7296061ca591f4bea5c8ee9665"}
Oct 30 23:50:29.229939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:29.229780    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4" event={"ID":"61c076ae-c401-4b0a-b3e6-d358bf9dd00c","Type":"ContainerStarted","Data":"b92301ab7b49e7234452aba7f68ada5a94e3a9645db3504ac094b551fe5de1d2"}
Oct 30 23:50:29.265761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:29.265710    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4" podStartSLOduration=2.265695667 podStartE2EDuration="2.265695667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:29.262272648 +0000 UTC m=+1888.188910231" watchObservedRunningTime="2026-10-30 23:50:29.265695667 +0000 UTC m=+1888.192333249"
Oct 30 23:50:29.925259 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:29.925224    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:37.191724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.190022    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"]
Oct 30 23:50:37.197875 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.197849    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.216809 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.216756    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7b0299-openshift-default-dockercfg-cq9kk\""
Oct 30 23:50:37.230219 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.230189    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"]
Oct 30 23:50:37.237686 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.237656    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.237852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.237731    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0016f269-b213-41a9-b78e-5725aac9de1e-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.237852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.237815    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0016f269-b213-41a9-b78e-5725aac9de1e-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.237972 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.237860    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.237972 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.237891    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.237972 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.237943    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.238131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.238080    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6lwn\" (UniqueName: \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-kube-api-access-g6lwn\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.238187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.238131    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.238187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.238169    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.341242    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6lwn\" (UniqueName: \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-kube-api-access-g6lwn\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.341325    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.341384    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.341510    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.341581    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0016f269-b213-41a9-b78e-5725aac9de1e-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.341684    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0016f269-b213-41a9-b78e-5725aac9de1e-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.341740    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.341776    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.341831    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.342410    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.342984    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.343255    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.345683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.344842    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0016f269-b213-41a9-b78e-5725aac9de1e-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.346526 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.345840    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.357403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.351293    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.384076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.383636    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6lwn\" (UniqueName: \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-kube-api-access-g6lwn\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.390482 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.385950    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0016f269-b213-41a9-b78e-5725aac9de1e-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.400441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.391900    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.541395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.536746    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:37.934176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.933012    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:37.954346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:37.954317    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:38.224578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:38.224547    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"]
Oct 30 23:50:38.225102 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:50:38.225078    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0016f269_b213_41a9_b78e_5725aac9de1e.slice/crio-645c2f36677aec0d1272d0e8c0755654045032e4761b857ea8c56e0d4080933a WatchSource:0}: Error finding container 645c2f36677aec0d1272d0e8c0755654045032e4761b857ea8c56e0d4080933a: Status 404 returned error can't find the container with id 645c2f36677aec0d1272d0e8c0755654045032e4761b857ea8c56e0d4080933a
Oct 30 23:50:38.228653 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:38.228621    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:38.228730 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:38.228686    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:38.228765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:38.228734    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:38.466886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:38.466846    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" event={"ID":"0016f269-b213-41a9-b78e-5725aac9de1e","Type":"ContainerStarted","Data":"8d54a0292141e6f0cf7d6ab2628ca5d7091a02cbb1baf23f2811be200a5d61e7"}
Oct 30 23:50:38.467082 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:38.466925    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" event={"ID":"0016f269-b213-41a9-b78e-5725aac9de1e","Type":"ContainerStarted","Data":"645c2f36677aec0d1272d0e8c0755654045032e4761b857ea8c56e0d4080933a"}
Oct 30 23:50:38.484323 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:38.484247    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:50:38.503393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:38.503065    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" podStartSLOduration=1.503045904 podStartE2EDuration="1.503045904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:38.491695617 +0000 UTC m=+1897.418333206" watchObservedRunningTime="2026-10-30 23:50:38.503045904 +0000 UTC m=+1897.429683477"
Oct 30 23:50:38.537770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:38.537737    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:39.543727 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:39.543698    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:40.045995 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.045963    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"]
Oct 30 23:50:40.055395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.052160    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.072216 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.072175    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"]
Oct 30 23:50:40.073773 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.073751    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-23f75e-openshift-default-dockercfg-ngphj\""
Oct 30 23:50:40.080363 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.080336    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0829125e-85cf-4a47-af4a-f5c820d7c513-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.080549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.080419    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.080549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.080457    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.080549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.080490    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwjw\" (UniqueName: \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-kube-api-access-xxwjw\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.080549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.080521    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.080763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.080569    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.080763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.080725    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.080763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.080754    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.080922 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.080834    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.182311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.182274    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.182520 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.182323    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.182520 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.182409    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.182662 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.182568    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0829125e-85cf-4a47-af4a-f5c820d7c513-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.182662 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.182624    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.182769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.182660    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.182769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.182696    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwjw\" (UniqueName: \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-kube-api-access-xxwjw\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.182769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.182729    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.183000 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.182787    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.183000 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.182790    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.183121 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.183105    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.183208 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.183185    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.183424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.183400    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.183925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.183898    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0829125e-85cf-4a47-af4a-f5c820d7c513-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.186051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.186024    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.186159 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.186123    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.196164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.196138    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.196494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.196470    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwjw\" (UniqueName: \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-kube-api-access-xxwjw\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.382450 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.381725    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:40.494626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.494579    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:40.496264 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.496240    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:50:40.770237 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:50:40.770210    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0829125e_85cf_4a47_af4a_f5c820d7c513.slice/crio-58fd5c290f0dd355e1b5ecacbd7d36c4948f16d8491bc2c2520f8cc080da6102 WatchSource:0}: Error finding container 58fd5c290f0dd355e1b5ecacbd7d36c4948f16d8491bc2c2520f8cc080da6102: Status 404 returned error can't find the container with id 58fd5c290f0dd355e1b5ecacbd7d36c4948f16d8491bc2c2520f8cc080da6102
Oct 30 23:50:40.774167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.774128    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:40.774272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.774209    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:40.774272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.774239    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:40.774352 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:40.774269    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"]
Oct 30 23:50:41.506874 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:41.506834    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" event={"ID":"0829125e-85cf-4a47-af4a-f5c820d7c513","Type":"ContainerStarted","Data":"e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38"}
Oct 30 23:50:41.507081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:41.506907    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" event={"ID":"0829125e-85cf-4a47-af4a-f5c820d7c513","Type":"ContainerStarted","Data":"58fd5c290f0dd355e1b5ecacbd7d36c4948f16d8491bc2c2520f8cc080da6102"}
Oct 30 23:50:41.535418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:41.535357    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" podStartSLOduration=1.535342398 podStartE2EDuration="1.535342398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:41.53049568 +0000 UTC m=+1900.457133261" watchObservedRunningTime="2026-10-30 23:50:41.535342398 +0000 UTC m=+1900.461979979"
Oct 30 23:50:42.388615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:42.388587    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:42.480774 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:42.392493    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 23:50:42.480774 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:42.478566    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 23:50:43.250870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.250837    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"]
Oct 30 23:50:43.256231 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.256180    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.340501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.340457    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"]
Oct 30 23:50:43.340915 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.340894    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-23f75e\"/\"test-model-23f75e-kserve-self-signed-certs\""
Oct 30 23:50:43.341044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.341029    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-23f75e\"/\"kube-root-ca.crt\""
Oct 30 23:50:43.341116 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.341103    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-23f75e\"/\"openshift-service-ca.crt\""
Oct 30 23:50:43.341480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.341459    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-23f75e\"/\"default-dockercfg-qzkjt\""
Oct 30 23:50:43.356520 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.356485    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-home\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.356870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.356836    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-dshm\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.357086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.357062    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-tmp-dir\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.357333 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.357315    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a721584-d8fe-4b76-b899-774cb70760d8-tls-certs\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.357489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.357472    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-model-cache\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.357583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.357568    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9slr\" (UniqueName: \"kubernetes.io/projected/5a721584-d8fe-4b76-b899-774cb70760d8-kube-api-access-c9slr\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.391421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.386760    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 23:50:43.391421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.386829    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 23:50:43.473421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.473332    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a721584-d8fe-4b76-b899-774cb70760d8-tls-certs\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.473608 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.473445    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-model-cache\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.473608 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.473500    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9slr\" (UniqueName: \"kubernetes.io/projected/5a721584-d8fe-4b76-b899-774cb70760d8-kube-api-access-c9slr\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.473608 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.473558    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-home\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.473812 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.473651    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-dshm\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.473812 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.473692    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-tmp-dir\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.474167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.474136    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-tmp-dir\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.494421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.481085    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-model-cache\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.494421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.481344    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-home\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.593395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.593286    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a721584-d8fe-4b76-b899-774cb70760d8-tls-certs\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.616392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.605860    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-dshm\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.616392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.610009    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9slr\" (UniqueName: \"kubernetes.io/projected/5a721584-d8fe-4b76-b899-774cb70760d8-kube-api-access-c9slr\") pod \"test-model-23f75e-kserve-74fc74f67f-hlh9p\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") " pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:43.653394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:43.644862    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:44.226622 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:44.226467    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"]
Oct 30 23:50:44.391394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:44.390773    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 23:50:44.391394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:44.390837    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 23:50:44.923158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:44.923121    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" event={"ID":"5a721584-d8fe-4b76-b899-774cb70760d8","Type":"ContainerStarted","Data":"6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a"}
Oct 30 23:50:44.923363 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:44.923173    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" event={"ID":"5a721584-d8fe-4b76-b899-774cb70760d8","Type":"ContainerStarted","Data":"ddf21b7bda35688a33f16874915c5355945730a946e8b64ed809c64c7bced4ec"}
Oct 30 23:50:44.943215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:44.943164    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" podStartSLOduration=1.943150624 podStartE2EDuration="1.943150624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:44.942277469 +0000 UTC m=+1903.868915048" watchObservedRunningTime="2026-10-30 23:50:44.943150624 +0000 UTC m=+1903.869788213"
Oct 30 23:50:45.389524 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:45.389479    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 23:50:45.389815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:45.389784    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 23:50:45.652442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:45.651533    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:46.384398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:46.383892    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 23:50:46.384398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:46.383968    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 23:50:46.832395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:46.831519    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"]
Oct 30 23:50:46.832395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:46.831807    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy" containerID="cri-o://8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e" gracePeriod=30
Oct 30 23:50:47.398398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:47.383821    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 23:50:47.398398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:47.383884    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 23:50:48.384344 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:48.384149    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 23:50:48.384344 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:48.384230    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 23:50:49.390163 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:49.390128    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:49.390694 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:49.390558    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:49.391592 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:49.391570    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:50:50.117382 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:50.117330    4555 patch_prober.go:29] interesting pod/e2e-mt-6a162557-openshift-default-8677879c89-nprf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:50:50.117545 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:50.117414    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:50:51.130670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.130632    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"]
Oct 30 23:50:51.135814 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.135794    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.143409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.143357    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1bc837-openshift-default-dockercfg-5htq6\""
Oct 30 23:50:51.151575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.151548    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"]
Oct 30 23:50:51.285411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.285317    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-podinfo\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.285623 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.285506    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-envoy\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.285730 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.285706    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75f9k\" (UniqueName: \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-kube-api-access-75f9k\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.285791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.285754    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-data\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.285900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.285880    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-credential-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.285959 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.285928    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.286022 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.286006    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-token\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.286083 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.286064    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istiod-ca-cert\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.286219 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.286196    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-certs\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.396452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.392145    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75f9k\" (UniqueName: \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-kube-api-access-75f9k\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.396452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.392200    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-data\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.396452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.392276    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-credential-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.396452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.392326    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.399909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.399868    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-token\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.400247 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.400223    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istiod-ca-cert\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.400483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.400465    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-certs\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.400681 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.400666    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-podinfo\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.400859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.400839    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-envoy\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.408428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.402646    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-data\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.408659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.408540    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-certs\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.409190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.409159    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istiod-ca-cert\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.422191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.409904    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-credential-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.422478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.410197    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-socket\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.430391 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.428404    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-podinfo\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.446397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.433339    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75f9k\" (UniqueName: \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-kube-api-access-75f9k\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.446692 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.441972    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-token\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.469083 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.468713    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-envoy\") pod \"e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") " pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:51.762253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:51.762164    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:52.062823 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:50:52.062794    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e0aef4_ac27_427f_85ac_f8c4fe2ba531.slice/crio-4ec7df2a77f5021c510f156e15df99311a88c3688007e1eb6a610c928ee34dc7 WatchSource:0}: Error finding container 4ec7df2a77f5021c510f156e15df99311a88c3688007e1eb6a610c928ee34dc7: Status 404 returned error can't find the container with id 4ec7df2a77f5021c510f156e15df99311a88c3688007e1eb6a610c928ee34dc7
Oct 30 23:50:52.063279 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.063252    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"]
Oct 30 23:50:52.066922 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.066881    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:52.067041 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.067024    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:52.067103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.067076    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:52.152691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.152659    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:50:52.239217 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239176    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-certs\") pod \"277a6da0-1746-45de-808f-e9a8b3a26ce3\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") "
Oct 30 23:50:52.239402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239226    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277a6da0-1746-45de-808f-e9a8b3a26ce3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a6da0-1746-45de-808f-e9a8b3a26ce3-istiod-ca-cert\") pod \"277a6da0-1746-45de-808f-e9a8b3a26ce3\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") "
Oct 30 23:50:52.239402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239266    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-credential-socket\") pod \"277a6da0-1746-45de-808f-e9a8b3a26ce3\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") "
Oct 30 23:50:52.239537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239411    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-data\") pod \"277a6da0-1746-45de-808f-e9a8b3a26ce3\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") "
Oct 30 23:50:52.239537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239453    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-podinfo\") pod \"277a6da0-1746-45de-808f-e9a8b3a26ce3\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") "
Oct 30 23:50:52.239537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239502    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-kube-api-access-8w9bj\" (UniqueName: \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-kube-api-access-8w9bj\") pod \"277a6da0-1746-45de-808f-e9a8b3a26ce3\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") "
Oct 30 23:50:52.239682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239535    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-certs" pod "277a6da0-1746-45de-808f-e9a8b3a26ce3" (UID: "277a6da0-1746-45de-808f-e9a8b3a26ce3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:50:52.239736 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239677    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-credential-socket" pod "277a6da0-1746-45de-808f-e9a8b3a26ce3" (UID: "277a6da0-1746-45de-808f-e9a8b3a26ce3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:50:52.239789 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239763    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-data" pod "277a6da0-1746-45de-808f-e9a8b3a26ce3" (UID: "277a6da0-1746-45de-808f-e9a8b3a26ce3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:50:52.239843 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239790    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-token\" (UniqueName: \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-token\") pod \"277a6da0-1746-45de-808f-e9a8b3a26ce3\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") "
Oct 30 23:50:52.239898 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239866    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-socket\") pod \"277a6da0-1746-45de-808f-e9a8b3a26ce3\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") "
Oct 30 23:50:52.239980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.239962    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-envoy\") pod \"277a6da0-1746-45de-808f-e9a8b3a26ce3\" (UID: \"277a6da0-1746-45de-808f-e9a8b3a26ce3\") "
Oct 30 23:50:52.240759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.240733    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-socket" pod "277a6da0-1746-45de-808f-e9a8b3a26ce3" (UID: "277a6da0-1746-45de-808f-e9a8b3a26ce3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:50:52.241013 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.240983    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:52.241098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.241029    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:52.241098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.241046    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:52.241098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.241061    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:52.241504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.241458    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277a6da0-1746-45de-808f-e9a8b3a26ce3-istiod-ca-cert" pod "277a6da0-1746-45de-808f-e9a8b3a26ce3" (UID: "277a6da0-1746-45de-808f-e9a8b3a26ce3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:50:52.242670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.242649    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-token" pod "277a6da0-1746-45de-808f-e9a8b3a26ce3" (UID: "277a6da0-1746-45de-808f-e9a8b3a26ce3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:50:52.243607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.243577    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-kube-api-access-8w9bj" pod "277a6da0-1746-45de-808f-e9a8b3a26ce3" (UID: "277a6da0-1746-45de-808f-e9a8b3a26ce3"). InnerVolumeSpecName "kube-api-access-8w9bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:50:52.243607 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.243593    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-envoy" pod "277a6da0-1746-45de-808f-e9a8b3a26ce3" (UID: "277a6da0-1746-45de-808f-e9a8b3a26ce3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:50:52.246412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.246346    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-podinfo" pod "277a6da0-1746-45de-808f-e9a8b3a26ce3" (UID: "277a6da0-1746-45de-808f-e9a8b3a26ce3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:50:52.342761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.342673    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:52.342761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.342704    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w9bj\" (UniqueName: \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-kube-api-access-8w9bj\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:52.342761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.342718    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:52.342761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.342738    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a6da0-1746-45de-808f-e9a8b3a26ce3-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:52.342761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.342752    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a6da0-1746-45de-808f-e9a8b3a26ce3-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:52.354772 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.354730    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" event={"ID":"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531","Type":"ContainerStarted","Data":"7c168374d46d34653ef98ca3fd60fc14dfe3eb2207e8d2b6c6aaafee39407e17"}
Oct 30 23:50:52.354949 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.354846    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" event={"ID":"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531","Type":"ContainerStarted","Data":"4ec7df2a77f5021c510f156e15df99311a88c3688007e1eb6a610c928ee34dc7"}
Oct 30 23:50:52.357338 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.357308    4555 generic.go:356] "Generic (PLEG): container finished" podID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerID="8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e" exitCode=0
Oct 30 23:50:52.357513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.357481    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" event={"ID":"277a6da0-1746-45de-808f-e9a8b3a26ce3","Type":"ContainerDied","Data":"8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e"}
Oct 30 23:50:52.357574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.357526    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6" event={"ID":"277a6da0-1746-45de-808f-e9a8b3a26ce3","Type":"ContainerDied","Data":"4afcf0155f61f4e2bb6bc30a420147d296f931a79380ba64f966db57daaf7651"}
Oct 30 23:50:52.357574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.357555    4555 scope.go:122] "RemoveContainer" containerID="8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e"
Oct 30 23:50:52.357757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.357743    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"
Oct 30 23:50:52.377174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.377149    4555 scope.go:122] "RemoveContainer" containerID="8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e"
Oct 30 23:50:52.377629 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:50:52.377604    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e\": container with ID starting with 8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e not found: ID does not exist" containerID="8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e"
Oct 30 23:50:52.377735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.377640    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e"} err="failed to get container status \"8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e\": rpc error: code = NotFound desc = could not find container \"8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e\": container with ID starting with 8546f2a30fee1921bdcedc0cec64f85df6ea03cfd78434b306ebeb9afe2f8e0e not found: ID does not exist"
Oct 30 23:50:52.398753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.398700    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" podStartSLOduration=1.398686236 podStartE2EDuration="1.398686236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:52.390114425 +0000 UTC m=+1911.316752006" watchObservedRunningTime="2026-10-30 23:50:52.398686236 +0000 UTC m=+1911.325323816"
Oct 30 23:50:52.430168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.428942    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"]
Oct 30 23:50:52.431830 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.431808    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6a162557-openshift-default-8677879c89-nprf6"]
Oct 30 23:50:52.762555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:52.762462    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:53.211334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:53.211302    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" path="/var/lib/kubelet/pods/277a6da0-1746-45de-808f-e9a8b3a26ce3/volumes"
Oct 30 23:50:53.653796 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:53.646739    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:53.764297 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:53.764259    4555 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:50:53.764581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:53.764550    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 23:50:53.815020 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:53.814989    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:54.541795 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:54.541753    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:50:54.788428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:54.781601    4555 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:50:54.788428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:54.781679    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 23:50:55.128539 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.128348    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"]
Oct 30 23:50:55.129535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.129490    4555 cpu_manager.go:414] "containerMap: removing container" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy"
Oct 30 23:50:55.129535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.129514    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy"
Oct 30 23:50:55.129847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.129813    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="277a6da0-1746-45de-808f-e9a8b3a26ce3" containerName="istio-proxy"
Oct 30 23:50:55.146807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.135426    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.146807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.141160    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"]
Oct 30 23:50:55.278409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.275184    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqv5d\" (UniqueName: \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-kube-api-access-jqv5d\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.278409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.275337    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.278409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.275457    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41036c28-a137-4175-9b83-5d368911b3ac-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.278409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.275493    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.278409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.275546    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.278409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.275574    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.278409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.275663    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41036c28-a137-4175-9b83-5d368911b3ac-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.278409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.275780    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.278409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.275942    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.396714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.396450    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.396714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.396553    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41036c28-a137-4175-9b83-5d368911b3ac-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.396714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.396624    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.396932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.396738    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.396932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.396850    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqv5d\" (UniqueName: \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-kube-api-access-jqv5d\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.396997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.396915    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.396997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.396940    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.397070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.397050    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41036c28-a137-4175-9b83-5d368911b3ac-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.397106 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.397076    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.397140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.397127    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.397576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.397201    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-credential-socket\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.397576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.397527    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-certs\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.397927 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.397761    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41036c28-a137-4175-9b83-5d368911b3ac-istiod-ca-cert\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.397927 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.397896    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-data\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.399804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.399781    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41036c28-a137-4175-9b83-5d368911b3ac-istio-podinfo\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.399899 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.399862    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-envoy\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.410637 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.410480    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqv5d\" (UniqueName: \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-kube-api-access-jqv5d\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.413457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.413439    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-istio-token\") pod \"e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") " pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.512729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.512696    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:55.786739 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.786590    4555 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:50:55.787262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.787184    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 23:50:55.845976 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:50:55.845924    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41036c28_a137_4175_9b83_5d368911b3ac.slice/crio-9af77714b52562101a577ffc5c6d6f96cdfb3c8750ca887782ecb36859630bd0 WatchSource:0}: Error finding container 9af77714b52562101a577ffc5c6d6f96cdfb3c8750ca887782ecb36859630bd0: Status 404 returned error can't find the container with id 9af77714b52562101a577ffc5c6d6f96cdfb3c8750ca887782ecb36859630bd0
Oct 30 23:50:55.847446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.847419    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"]
Oct 30 23:50:55.849306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.849270    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:55.849446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.849353    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:55.849516 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:55.849443    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:50:56.449812 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:56.449776    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" event={"ID":"41036c28-a137-4175-9b83-5d368911b3ac","Type":"ContainerStarted","Data":"8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d"}
Oct 30 23:50:56.449997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:56.449838    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" event={"ID":"41036c28-a137-4175-9b83-5d368911b3ac","Type":"ContainerStarted","Data":"9af77714b52562101a577ffc5c6d6f96cdfb3c8750ca887782ecb36859630bd0"}
Oct 30 23:50:56.513721 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:56.513682    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:50:56.515840 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:56.515568    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 23:50:56.515840 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:56.515618    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 23:50:56.804395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:56.803516    4555 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:50:56.804395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:56.803584    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 23:50:57.459271 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:50:57.459233    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:50:57.514395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:57.513920    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 23:50:57.514395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:57.513980    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 23:50:57.773459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:57.765779    4555 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:50:57.773459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:57.765839    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 23:50:58.545510 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:58.534070    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 23:50:58.545510 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:58.534132    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 23:50:58.764396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:58.763791    4555 patch_prober.go:29] interesting pod/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:50:58.764396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:58.763856    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 23:50:59.513876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:59.513825    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 23:50:59.514052 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:59.513927    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 23:50:59.784395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:59.783648    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:59.788390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:59.785396    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:59.792735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:59.792708    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:50:59.821094 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:50:59.821039    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" podStartSLOduration=4.821020831 podStartE2EDuration="4.821020831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:56.497314107 +0000 UTC m=+1915.423951689" watchObservedRunningTime="2026-10-30 23:50:59.821020831 +0000 UTC m=+1918.747658404"
Oct 30 23:51:00.513497 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:00.513462    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 23:51:00.513695 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:00.513523    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 23:51:01.513604 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:01.513568    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 23:51:01.514102 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:01.513613    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 23:51:02.517090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:02.514789    4555 patch_prober.go:29] interesting pod/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 23:51:02.517090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:02.514863    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 23:51:03.519554 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:03.519518    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:51:03.520031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:03.519975    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:51:03.520895 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:03.520874    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:51:08.843449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.843413    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"]
Oct 30 23:51:08.851860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.851835    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.856272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.856220    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"]
Oct 30 23:51:08.857948 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.857932    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7fc6e9\"/\"test-model-7fc6e9-kserve-self-signed-certs\""
Oct 30 23:51:08.859015 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.858995    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7fc6e9\"/\"openshift-service-ca.crt\""
Oct 30 23:51:08.859163 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.859137    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7fc6e9\"/\"kube-root-ca.crt\""
Oct 30 23:51:08.859234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.859043    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7fc6e9\"/\"default-dockercfg-zmc49\""
Oct 30 23:51:08.889409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.889355    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-home\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.889586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.889423    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-tmp-dir\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.889586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.889551    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2rl\" (UniqueName: \"kubernetes.io/projected/462a2480-06db-45c5-8fc5-b0e5b714cadb-kube-api-access-zv2rl\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.889703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.889660    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-dshm\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.889855 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.889836    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-model-cache\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.889946 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.889931    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462a2480-06db-45c5-8fc5-b0e5b714cadb-tls-certs\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.991150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.991107    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2rl\" (UniqueName: \"kubernetes.io/projected/462a2480-06db-45c5-8fc5-b0e5b714cadb-kube-api-access-zv2rl\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.991353 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.991247    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-dshm\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.991548 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.991525    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-model-cache\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.991845 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.991821    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462a2480-06db-45c5-8fc5-b0e5b714cadb-tls-certs\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.991976 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.991901    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-home\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.991976 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.991929    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-tmp-dir\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.992586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.992565    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-home\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.992804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.992771    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-tmp-dir\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.994717 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.994619    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-dshm\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.994879 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.991893    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-model-cache\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:08.997286 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:08.997223    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462a2480-06db-45c5-8fc5-b0e5b714cadb-tls-certs\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:09.086770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:09.086699    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2rl\" (UniqueName: \"kubernetes.io/projected/462a2480-06db-45c5-8fc5-b0e5b714cadb-kube-api-access-zv2rl\") pod \"test-model-7fc6e9-kserve-6687bc68c5-brcr4\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") " pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:09.267489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:09.248715    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:09.776353 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:09.776305    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"]
Oct 30 23:51:09.782518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:09.779159    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" event={"ID":"462a2480-06db-45c5-8fc5-b0e5b714cadb","Type":"ContainerStarted","Data":"bd8af71f937ddc096b145af3fb485fe63d5a5806f0408b752ae11cb8096ce71d"}
Oct 30 23:51:10.791994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:10.791951    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" event={"ID":"462a2480-06db-45c5-8fc5-b0e5b714cadb","Type":"ContainerStarted","Data":"f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847"}
Oct 30 23:51:10.815799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:10.815740    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" podStartSLOduration=2.815722779 podStartE2EDuration="2.815722779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:10.813879253 +0000 UTC m=+1929.740516835" watchObservedRunningTime="2026-10-30 23:51:10.815722779 +0000 UTC m=+1929.742360365"
Oct 30 23:51:11.031172 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:11.031142    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"]
Oct 30 23:51:11.038887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:11.038851    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e2ee664-bwxws"]
Oct 30 23:51:11.213350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:11.213274    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acec78f6-135c-4d2e-80e2-f2b70d70eb5c" path="/var/lib/kubelet/pods/acec78f6-135c-4d2e-80e2-f2b70d70eb5c/volumes"
Oct 30 23:51:11.251309 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:11.251280    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:12.026331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:12.026300    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"]
Oct 30 23:51:12.031776 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:12.031748    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ca1003-n6vfl"]
Oct 30 23:51:13.211911 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:13.211876    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24982555-2ef8-4285-8db9-d9b19f8ef63e" path="/var/lib/kubelet/pods/24982555-2ef8-4285-8db9-d9b19f8ef63e/volumes"
Oct 30 23:51:17.033194 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:17.033158    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"]
Oct 30 23:51:17.041979 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:17.041951    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4a201e-mm2sj"]
Oct 30 23:51:17.226735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:17.226699    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba" path="/var/lib/kubelet/pods/a6f77f84-b56f-4b99-a7c4-e51c7b7b09ba/volumes"
Oct 30 23:51:19.251473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:19.251442    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:19.265921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:19.265889    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:19.946810 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:19.946781    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:51:20.004636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:20.004597    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:51:23.366941 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:23.366704    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 23:51:23.496398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:23.470995    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4" podUID="61c076ae-c401-4b0a-b3e6-d358bf9dd00c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.146:8000/ready\": context deadline exceeded"
Oct 30 23:51:23.552968 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:23.531489    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:23.634116 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:23.634012    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 23:51:23.774394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:23.680124    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Oct 30 23:51:23.950808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:23.950686    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.152:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:23.968692 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:23.968643    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:24.024559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.024514    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:24.254150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.253267    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"]
Oct 30 23:51:24.280724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.280692    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.344584 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.344549    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f0817c\"/\"openshift-service-ca.crt\""
Oct 30 23:51:24.344782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.344663    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f0817c\"/\"test-model-f0817c-kserve-self-signed-certs\""
Oct 30 23:51:24.344782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.344736    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f0817c\"/\"kube-root-ca.crt\""
Oct 30 23:51:24.344909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.344818    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f0817c\"/\"default-dockercfg-hvr7v\""
Oct 30 23:51:24.385914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.381902    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-home\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.385914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.382032    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-model-cache\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.385914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.382266    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648f4553-fa26-444b-a2e4-a690b559990f-tls-certs\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.385914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.382391    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-dshm\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.385914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.382496    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-tmp-dir\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.385914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.382592    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brz44\" (UniqueName: \"kubernetes.io/projected/648f4553-fa26-444b-a2e4-a690b559990f-kube-api-access-brz44\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.388454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.388422    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"]
Oct 30 23:51:24.551394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.546609    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-home\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.551394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.546714    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-model-cache\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.551394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.546889    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648f4553-fa26-444b-a2e4-a690b559990f-tls-certs\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.551394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.546971    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-dshm\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.551394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.547061    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-tmp-dir\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.551394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.547145    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brz44\" (UniqueName: \"kubernetes.io/projected/648f4553-fa26-444b-a2e4-a690b559990f-kube-api-access-brz44\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.551394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.548031    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-home\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.551394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.548297    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-model-cache\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.586983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.586937    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-dshm\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.616983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.601695    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-tmp-dir\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.616983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.607416    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brz44\" (UniqueName: \"kubernetes.io/projected/648f4553-fa26-444b-a2e4-a690b559990f-kube-api-access-brz44\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.640456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.634449    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648f4553-fa26-444b-a2e4-a690b559990f-tls-certs\") pod \"test-model-f0817c-kserve-ff4d44fd9-gxz7v\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") " pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:24.700510 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:24.700478    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:25.130087 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:51:25.130029    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod648f4553_fa26_444b_a2e4_a690b559990f.slice/crio-dfb3ca12153962ff0c1c0c9b90b53963c896348a92206f12faa3c03dc3ed3957 WatchSource:0}: Error finding container dfb3ca12153962ff0c1c0c9b90b53963c896348a92206f12faa3c03dc3ed3957: Status 404 returned error can't find the container with id dfb3ca12153962ff0c1c0c9b90b53963c896348a92206f12faa3c03dc3ed3957
Oct 30 23:51:25.132755 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:25.132529    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"]
Oct 30 23:51:25.485893 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:25.485795    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v" event={"ID":"648f4553-fa26-444b-a2e4-a690b559990f","Type":"ContainerStarted","Data":"dfb3ca12153962ff0c1c0c9b90b53963c896348a92206f12faa3c03dc3ed3957"}
Oct 30 23:51:26.197395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:26.190946    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": context deadline exceeded"
Oct 30 23:51:26.450575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:26.450462    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:26.501395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:26.489103    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4" podUID="61c076ae-c401-4b0a-b3e6-d358bf9dd00c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.146:8000/ready\": context deadline exceeded"
Oct 30 23:51:26.555396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:26.530046    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:26.688454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:26.641198    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 23:51:26.765275 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:26.743433    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:26.867923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:26.867871    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v" event={"ID":"648f4553-fa26-444b-a2e4-a690b559990f","Type":"ContainerStarted","Data":"829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d"}
Oct 30 23:51:27.268395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.260069    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:27.457440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.454570    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:27.485406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.458050    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:51:27.485406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.458141    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:51:27.510944 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.508498    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:27.510944 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.509109    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:51:27.556983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.556906    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v" podStartSLOduration=4.55688418 podStartE2EDuration="4.55688418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:27.026878937 +0000 UTC m=+1945.953516541" watchObservedRunningTime="2026-10-30 23:51:27.55688418 +0000 UTC m=+1946.483521761"
Oct 30 23:51:27.594753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.586483    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:27.668601 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.668552    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:27.720866 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.714686    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:27.754394 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:51:27.741724    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:51:27.754394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.746142    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:27.779396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.773502    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:27.779396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.773956    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:27.779396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.774335    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:51:27.962618 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:27.962513    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Oct 30 23:51:28.116885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:28.028551    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.152:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:51:28.453397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:28.448964    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Oct 30 23:51:28.519396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:28.519046    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 23:51:28.594398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:28.593854    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:51:28.905407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:28.902779    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:51:30.100776 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.100740    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"]
Oct 30 23:51:30.115317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.112692    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.115993 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.115166    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"]
Oct 30 23:51:30.126705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.126610    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a63288\"/\"kube-root-ca.crt\""
Oct 30 23:51:30.126705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.126709    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a63288\"/\"test-model-a63288-kserve-self-signed-certs\""
Oct 30 23:51:30.127575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.126824    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a63288\"/\"default-dockercfg-blpgh\""
Oct 30 23:51:30.127575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.126853    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a63288\"/\"openshift-service-ca.crt\""
Oct 30 23:51:30.186745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.186704    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxw55\" (UniqueName: \"kubernetes.io/projected/0cbd6143-2e09-46e2-b701-f1e457c32d1f-kube-api-access-gxw55\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.186953 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.186827    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tmp-dir\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.186953 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.186857    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-home\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.186953 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.186910    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-model-cache\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.186953 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.186938    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tls-certs\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.187174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.186966    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-dshm\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.306397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.303643    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxw55\" (UniqueName: \"kubernetes.io/projected/0cbd6143-2e09-46e2-b701-f1e457c32d1f-kube-api-access-gxw55\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.306397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.303807    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tmp-dir\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.306397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.303836    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-home\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.306397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.303907    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-model-cache\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.306397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.303943    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tls-certs\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.306397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.303978    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-dshm\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.306397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.304816    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-home\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.306397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.306030    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tmp-dir\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.307633 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.307607    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-model-cache\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.346858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.346818    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-dshm\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.347849 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.347818    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tls-certs\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.358395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.358252    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxw55\" (UniqueName: \"kubernetes.io/projected/0cbd6143-2e09-46e2-b701-f1e457c32d1f-kube-api-access-gxw55\") pod \"test-model-a63288-kserve-5b8bfccbf5-pdxhm\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") " pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.439932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.439900    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:30.942887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:30.942831    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"]
Oct 30 23:51:30.945801 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:51:30.945726    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cbd6143_2e09_46e2_b701_f1e457c32d1f.slice/crio-38bacb6e48b2a475b40c96d5da9645f012898fa82ec2fdf6fabb486dc9a4e695 WatchSource:0}: Error finding container 38bacb6e48b2a475b40c96d5da9645f012898fa82ec2fdf6fabb486dc9a4e695: Status 404 returned error can't find the container with id 38bacb6e48b2a475b40c96d5da9645f012898fa82ec2fdf6fabb486dc9a4e695
Oct 30 23:51:31.440270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:31.440214    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm" event={"ID":"0cbd6143-2e09-46e2-b701-f1e457c32d1f","Type":"ContainerStarted","Data":"a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40"}
Oct 30 23:51:31.440819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:31.440278    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm" event={"ID":"0cbd6143-2e09-46e2-b701-f1e457c32d1f","Type":"ContainerStarted","Data":"38bacb6e48b2a475b40c96d5da9645f012898fa82ec2fdf6fabb486dc9a4e695"}
Oct 30 23:51:31.440819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:31.440416    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:31.462328 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:31.462280    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm" podStartSLOduration=1.462264576 podStartE2EDuration="1.462264576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:31.458888901 +0000 UTC m=+1950.385526485" watchObservedRunningTime="2026-10-30 23:51:31.462264576 +0000 UTC m=+1950.388902156"
Oct 30 23:51:34.728397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:34.715554    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:34.807675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:34.807153    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:34.837734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:34.837703    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:51:39.816898 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.816862    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"]
Oct 30 23:51:39.829664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.829632    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:39.829889 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.829866    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"]
Oct 30 23:51:39.839221 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.839194    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-32732d\"/\"test-model-32732d-kserve-self-signed-certs\""
Oct 30 23:51:39.840837 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.840587    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-32732d\"/\"default-dockercfg-wrmcf\""
Oct 30 23:51:39.891393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.890856    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-32732d\"/\"openshift-service-ca.crt\""
Oct 30 23:51:39.891393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.890971    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-32732d\"/\"kube-root-ca.crt\""
Oct 30 23:51:39.915493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.915448    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-home\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:39.915688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.915513    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-model-cache\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:39.915688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.915609    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tmp-dir\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:39.916190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.915920    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tls-certs\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:39.916190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.916006    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-dshm\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:39.916190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:39.916104    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg5mb\" (UniqueName: \"kubernetes.io/projected/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-kube-api-access-bg5mb\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.037403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.024777    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-home\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.037403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.024885    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-model-cache\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.037403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.024927    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tmp-dir\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.037403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.025275    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tls-certs\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.037403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.025362    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-dshm\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.037403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.025459    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg5mb\" (UniqueName: \"kubernetes.io/projected/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-kube-api-access-bg5mb\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.037403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.026278    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-home\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.037403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.026562    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-model-cache\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.037403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.026800    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tmp-dir\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.073280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.073165    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg5mb\" (UniqueName: \"kubernetes.io/projected/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-kube-api-access-bg5mb\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.073708 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.073684    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-dshm\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.074462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.074437    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tls-certs\") pod \"test-model-32732d-kserve-5ff7dbc97-6cp6l\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") " pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.157013 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.156695    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:40.443556 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.441609    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:40.479103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.474608    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:40.730578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.730542    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"]
Oct 30 23:51:40.879328 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.879301    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l" event={"ID":"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce","Type":"ContainerStarted","Data":"753cfe5f49df6f00174410082c0643880f9b7960ec5d4f2bb081483963fcd683"}
Oct 30 23:51:40.895746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:40.895703    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:51:41.908162 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:41.908119    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l" event={"ID":"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce","Type":"ContainerStarted","Data":"d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1"}
Oct 30 23:51:41.928990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:41.928932    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l" podStartSLOduration=2.928917648 podStartE2EDuration="2.928917648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:41.928301306 +0000 UTC m=+1960.854938891" watchObservedRunningTime="2026-10-30 23:51:41.928917648 +0000 UTC m=+1960.855555229"
Oct 30 23:51:42.158018 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:42.157985    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:46.527218 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.527177    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"]
Oct 30 23:51:46.533424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.533354    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.544750 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.544721    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-82d449\"/\"openshift-service-ca.crt\""
Oct 30 23:51:46.544946 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.544862    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-82d449\"/\"default-dockercfg-6rzvm\""
Oct 30 23:51:46.544946 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.544914    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-82d449\"/\"test-model-82d449-kserve-self-signed-certs\""
Oct 30 23:51:46.546586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.546422    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-82d449\"/\"kube-root-ca.crt\""
Oct 30 23:51:46.550891 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.550703    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"]
Oct 30 23:51:46.623422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.622362    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xn9\" (UniqueName: \"kubernetes.io/projected/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-kube-api-access-w9xn9\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.623422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.622474    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-home\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.623422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.622512    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-dshm\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.623422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.622636    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tmp-dir\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.623422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.622782    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-model-cache\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.623422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.622848    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tls-certs\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.724087 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.724031    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xn9\" (UniqueName: \"kubernetes.io/projected/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-kube-api-access-w9xn9\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.724306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.724103    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-home\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.724306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.724135    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-dshm\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.724306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.724225    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tmp-dir\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.724518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.724322    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-model-cache\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.724518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.724386    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tls-certs\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.724632 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.724607    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-home\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.724735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.724714    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tmp-dir\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.724989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.724956    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-model-cache\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.727105 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.727074    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-dshm\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.727309 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.727294    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tls-certs\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.777442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.771537    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xn9\" (UniqueName: \"kubernetes.io/projected/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-kube-api-access-w9xn9\") pod \"test-model-82d449-kserve-5498db5685-tbkhl\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") " pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:46.865007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:46.864710    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:47.483223 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:47.483196    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"]
Oct 30 23:51:47.487896 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:51:47.487868    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d0ac0c9_f4d4_46d4_81b5_b702bff7f7d1.slice/crio-2afca62b711331619d0ebefdb62ccb9408084d7e9cd5120540ae6ffb46aead1c WatchSource:0}: Error finding container 2afca62b711331619d0ebefdb62ccb9408084d7e9cd5120540ae6ffb46aead1c: Status 404 returned error can't find the container with id 2afca62b711331619d0ebefdb62ccb9408084d7e9cd5120540ae6ffb46aead1c
Oct 30 23:51:48.147870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:48.147831    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl" event={"ID":"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1","Type":"ContainerStarted","Data":"9edb46cbf5cf5cd35959db61f6ef0c03d5619455bb888d7bd75846ba1ad25a4d"}
Oct 30 23:51:48.147870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:48.147887    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl" event={"ID":"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1","Type":"ContainerStarted","Data":"2afca62b711331619d0ebefdb62ccb9408084d7e9cd5120540ae6ffb46aead1c"}
Oct 30 23:51:48.168204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:48.168149    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl" podStartSLOduration=2.16813549 podStartE2EDuration="2.16813549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:48.167587271 +0000 UTC m=+1967.094224843" watchObservedRunningTime="2026-10-30 23:51:48.16813549 +0000 UTC m=+1967.094773071"
Oct 30 23:51:48.865581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:48.865549    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:50.158453 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:50.158400    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:50.174252 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:50.174213    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:50.190725 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:50.190693    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:51:54.173720 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.173685    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"]
Oct 30 23:51:54.186393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.182709    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.204313 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.204281    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7b0299\"/\"openshift-service-ca.crt\""
Oct 30 23:51:54.204521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.204405    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7b0299\"/\"kube-root-ca.crt\""
Oct 30 23:51:54.204521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.204516    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7b0299\"/\"default-dockercfg-8pg2v\""
Oct 30 23:51:54.204648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.204600    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7b0299\"/\"test-model-7b0299-kserve-self-signed-certs\""
Oct 30 23:51:54.213199 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.213168    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"]
Oct 30 23:51:54.324403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.324341    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-tmp-dir\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.324861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.324832    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-dshm\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.325103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.325071    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-model-cache\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.325192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.325164    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-home\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.325283 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.325263    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f832f680-ca9d-434b-ac35-3c9c47105f8e-tls-certs\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.325488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.325467    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmmqt\" (UniqueName: \"kubernetes.io/projected/f832f680-ca9d-434b-ac35-3c9c47105f8e-kube-api-access-jmmqt\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.432449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.427865    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-dshm\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.432449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.427921    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-model-cache\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.432449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.427969    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-home\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.432449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.428055    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f832f680-ca9d-434b-ac35-3c9c47105f8e-tls-certs\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.432449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.428151    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmmqt\" (UniqueName: \"kubernetes.io/projected/f832f680-ca9d-434b-ac35-3c9c47105f8e-kube-api-access-jmmqt\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.432449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.428241    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-tmp-dir\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.432449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.428856    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-tmp-dir\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.432961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.432559    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-home\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.432961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.432843    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-model-cache\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.448783 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.448741    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-dshm\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.456165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.456134    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmmqt\" (UniqueName: \"kubernetes.io/projected/f832f680-ca9d-434b-ac35-3c9c47105f8e-kube-api-access-jmmqt\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.489773 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.486625    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f832f680-ca9d-434b-ac35-3c9c47105f8e-tls-certs\") pod \"test-model-7b0299-kserve-64c4b49cd9-kkxmt\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") " pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:54.540628 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:54.540597    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:55.060059 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:51:55.060026    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf832f680_ca9d_434b_ac35_3c9c47105f8e.slice/crio-e14fe55361956b9b1ca50bf02ee43b11465585d3f2b05319f26f576ca82b72cb WatchSource:0}: Error finding container e14fe55361956b9b1ca50bf02ee43b11465585d3f2b05319f26f576ca82b72cb: Status 404 returned error can't find the container with id e14fe55361956b9b1ca50bf02ee43b11465585d3f2b05319f26f576ca82b72cb
Oct 30 23:51:55.061936 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:55.061904    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"]
Oct 30 23:51:55.626396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:55.620027    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt" event={"ID":"f832f680-ca9d-434b-ac35-3c9c47105f8e","Type":"ContainerStarted","Data":"6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e"}
Oct 30 23:51:55.626396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:55.620081    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt" event={"ID":"f832f680-ca9d-434b-ac35-3c9c47105f8e","Type":"ContainerStarted","Data":"e14fe55361956b9b1ca50bf02ee43b11465585d3f2b05319f26f576ca82b72cb"}
Oct 30 23:51:55.678395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:55.673594    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt" podStartSLOduration=1.673571074 podStartE2EDuration="1.673571074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:55.650757738 +0000 UTC m=+1974.577395319" watchObservedRunningTime="2026-10-30 23:51:55.673571074 +0000 UTC m=+1974.600208656"
Oct 30 23:51:56.543747 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:56.543715    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:51:56.866863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:56.865839    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:56.902151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:56.902117    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:51:57.490398 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:51:57.484606    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:51:57.652892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:51:57.652863    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:52:04.542533 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:04.542495    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:52:04.558095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:04.558063    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:52:04.798325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:04.798246    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:52:07.817719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:07.817673    4555 generic.go:356] "Generic (PLEG): container finished" podID="0016f269-b213-41a9-b78e-5725aac9de1e" containerID="8d54a0292141e6f0cf7d6ab2628ca5d7091a02cbb1baf23f2811be200a5d61e7" exitCode=0
Oct 30 23:52:07.818203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:07.817745    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" event={"ID":"0016f269-b213-41a9-b78e-5725aac9de1e","Type":"ContainerDied","Data":"8d54a0292141e6f0cf7d6ab2628ca5d7091a02cbb1baf23f2811be200a5d61e7"}
Oct 30 23:52:07.818319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:07.818299    4555 scope.go:122] "RemoveContainer" containerID="8d54a0292141e6f0cf7d6ab2628ca5d7091a02cbb1baf23f2811be200a5d61e7"
Oct 30 23:52:07.820585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:07.820553    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:07.820682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:07.820628    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:07.820682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:07.820662    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:08.829504 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:08.829466    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" event={"ID":"0016f269-b213-41a9-b78e-5725aac9de1e","Type":"ContainerStarted","Data":"e23079989e9cb653203eef06c3a11a54a11b205148409368bd4a7a8b5a74f4c2"}
Oct 30 23:52:09.537291 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:09.537257    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:52:09.538260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:09.538227    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:52:09.538394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:09.538277    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:52:10.494967 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:10.494935    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:52:10.537660 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:10.537631    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:52:10.537819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:10.537696    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:52:11.538522 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:11.538193    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:52:11.538522 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:11.538254    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:52:12.537498 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:12.537462    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:52:12.537702 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:12.537517    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:52:13.537357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:13.537327    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:52:13.537858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:13.537399    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:52:14.543054 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:14.543019    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:52:14.544194 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:14.544169    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:52:15.491213 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.491178    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"]
Oct 30 23:52:15.495200 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.495179    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.502754 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.502724    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a63288-openshift-default-dockercfg-tdk54\""
Oct 30 23:52:15.511014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.510978    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"]
Oct 30 23:52:15.575157 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.575121    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-podinfo\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.575723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.575264    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-credential-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.575723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.575318    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istiod-ca-cert\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.575723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.575575    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.575723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.575600    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-data\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.575723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.575718    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-envoy\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.576001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.575771    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-certs\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.576001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.575852    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-token\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.576001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.575915    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vd8l\" (UniqueName: \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-kube-api-access-7vd8l\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.677562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.677523    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-podinfo\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.677761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.677613    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-credential-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.677761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.677641    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istiod-ca-cert\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.677886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.677846    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.677886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.677871    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-data\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.678021 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.677990    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-envoy\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.678073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.678048    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-certs\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.678129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.678066    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-credential-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.678183 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.678127    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-token\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.678236 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.678190    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vd8l\" (UniqueName: \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-kube-api-access-7vd8l\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.678236 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.678219    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-socket\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.678343 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.678302    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-data\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.678878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.678853    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istiod-ca-cert\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.679024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.678857    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-certs\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.680559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.680538    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-podinfo\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.681108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.681090    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-envoy\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.690948 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.690923    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vd8l\" (UniqueName: \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-kube-api-access-7vd8l\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.691302 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.691282    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-token\") pod \"e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") " pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.813308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.813270    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"]
Oct 30 23:52:15.823585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.822874    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:15.830964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.829580    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"]
Oct 30 23:52:15.830964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.829742    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.838890 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.835242    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2ccd42-openshift-default-dockercfg-qcc6m\""
Oct 30 23:52:15.890224 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.890191    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92725345-6154-47a7-847c-49feff85cdc7-istio-podinfo\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.890647 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.890627    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-envoy\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.890846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.890828    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-credential-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.891122 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.891104    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.891288 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.891274    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92725345-6154-47a7-847c-49feff85cdc7-istiod-ca-cert\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.891488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.891472    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-data\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.891743 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.891726    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2wv5\" (UniqueName: \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-kube-api-access-t2wv5\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.891931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.891912    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-istio-token\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.892107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.892087    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-certs\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.994635    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92725345-6154-47a7-847c-49feff85cdc7-istiod-ca-cert\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.994726    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-data\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.994779    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2wv5\" (UniqueName: \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-kube-api-access-t2wv5\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.994824    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-istio-token\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.994858    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-certs\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.995021    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92725345-6154-47a7-847c-49feff85cdc7-istio-podinfo\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.995152    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-envoy\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.995193    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-credential-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.995323    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.995819    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.996454    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92725345-6154-47a7-847c-49feff85cdc7-istiod-ca-cert\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.996867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.996708    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-certs\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.997651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.997020    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-data\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:15.998286 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:15.998239    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-credential-socket\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:16.029597 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.029531    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92725345-6154-47a7-847c-49feff85cdc7-istio-podinfo\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:16.032216 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.031911    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-envoy\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:16.032216 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.032167    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2wv5\" (UniqueName: \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-kube-api-access-t2wv5\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:16.033590 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.033549    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-istio-token\") pod \"e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") " pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:16.220713 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:52:16.220683    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df9f953_d9bb_4fc1_b335_8548c1b4a1f3.slice/crio-ee1713667a0e7e2f5a069ca212073f46a0d6a62cae24d8746d3923dba6880140 WatchSource:0}: Error finding container ee1713667a0e7e2f5a069ca212073f46a0d6a62cae24d8746d3923dba6880140: Status 404 returned error can't find the container with id ee1713667a0e7e2f5a069ca212073f46a0d6a62cae24d8746d3923dba6880140
Oct 30 23:52:16.221870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.221841    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"]
Oct 30 23:52:16.223828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.223796    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:16.223900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.223874    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:16.223936 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.223904    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:16.224524 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.224504    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:16.525392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.525199    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"]
Oct 30 23:52:16.594606 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.585190    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:16.612157 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.595418    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:16.612157 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.595527    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:16.994241 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.994201    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" event={"ID":"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3","Type":"ContainerStarted","Data":"aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713"}
Oct 30 23:52:16.994482 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.994445    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" event={"ID":"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3","Type":"ContainerStarted","Data":"ee1713667a0e7e2f5a069ca212073f46a0d6a62cae24d8746d3923dba6880140"}
Oct 30 23:52:16.996251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.996224    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" event={"ID":"92725345-6154-47a7-847c-49feff85cdc7","Type":"ContainerStarted","Data":"12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5"}
Oct 30 23:52:16.996411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:16.996280    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" event={"ID":"92725345-6154-47a7-847c-49feff85cdc7","Type":"ContainerStarted","Data":"62cbb9167089e4bd3a3a5e54e83ee694bcaf05dc74a29f18f3a48b14279dc742"}
Oct 30 23:52:17.017820 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:17.017756    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" podStartSLOduration=2.017736559 podStartE2EDuration="2.017736559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:17.016730926 +0000 UTC m=+1995.943368508" watchObservedRunningTime="2026-10-30 23:52:17.017736559 +0000 UTC m=+1995.944374141"
Oct 30 23:52:17.042033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:17.041968    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" podStartSLOduration=2.041950312 podStartE2EDuration="2.041950312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:17.038937959 +0000 UTC m=+1995.965575540" watchObservedRunningTime="2026-10-30 23:52:17.041950312 +0000 UTC m=+1995.968587896"
Oct 30 23:52:17.225047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:17.224956    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:17.226386 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:17.226350    4555 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 23:52:17.226485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:17.226419    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 23:52:17.831395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:17.829266    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:17.831395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:17.829476    4555 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:52:17.831395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:17.829526    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:52:18.227869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:18.227772    4555 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 23:52:18.228146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:18.228114    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 23:52:18.831934 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:18.831839    4555 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:52:18.832622 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:18.832579    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:52:19.230516 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:19.227898    4555 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 23:52:19.230516 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:19.227965    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 23:52:19.829014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:19.828963    4555 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:52:19.829331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:19.829286    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:52:20.236470 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:20.234645    4555 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 23:52:20.236470 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:20.234713    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 23:52:20.842405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:20.841113    4555 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:52:20.842405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:20.841251    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:52:21.227888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:21.227789    4555 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 23:52:21.233507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:21.233462    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 23:52:21.826841 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:21.826802    4555 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:52:21.827503 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:21.827470    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:52:22.228854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:22.228760    4555 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 23:52:22.229161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:22.229129    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 23:52:22.826858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:22.826821    4555 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:52:22.827505 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:22.827471    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:52:23.226899 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:23.226813    4555 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 23:52:23.227210 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:23.227164    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 23:52:24.033395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:24.001616    4555 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:52:24.033395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:24.001699    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:52:24.268397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:24.245207    4555 patch_prober.go:29] interesting pod/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 23:52:24.268397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:24.245274    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 23:52:24.582842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:24.582525    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:52:24.827239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:24.827201    4555 patch_prober.go:29] interesting pod/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:52:24.827461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:24.827267    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:52:25.291772 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:25.291737    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:25.292670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:25.292616    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:25.294388 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:25.294344    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:52:25.831026 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:25.830998    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:25.831521 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:25.831493    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:25.832250 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:25.832218    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:52:27.448347 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:52:27.448309    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:52:35.163762 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:35.163726    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:52:37.535397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.526421    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"]
Oct 30 23:52:37.548397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.539692    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.614468 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.614431    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"]
Oct 30 23:52:37.662432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.662396    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.662818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.662784    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.663018 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.663002    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.663119 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.663106    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.663285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.663271    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.663410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.663395    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.663591 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.663577    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.663743 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.663730    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.663907 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.663893    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpbk\" (UniqueName: \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-kube-api-access-wfpbk\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.703676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.703640    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"]
Oct 30 23:52:37.709476 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.703986    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:37.783401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.769625    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.857448    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.857555    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.857643    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.857730    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpbk\" (UniqueName: \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-kube-api-access-wfpbk\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.857940    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.858050    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.858108    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.858130    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.858657    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.858869    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.859460    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:37.908676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.859746    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:38.045917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:37.976155    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:38.058327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.058289    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.058617 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.058594    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.058791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.058775    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.058949 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.058928    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e0b9a5-206c-4589-a9dc-4b4f46620605-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.059248 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.059226    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.059803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.059542    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.059803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.059621    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:38.059803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.059659    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td47r\" (UniqueName: \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-kube-api-access-td47r\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.060035 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.059914    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.060035 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.059943    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.099127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.098529    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpbk\" (UniqueName: \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-kube-api-access-wfpbk\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:38.102536 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.102509    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:38.122677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.106091    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:38.160438 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.157506    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"]
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.162231    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td47r\" (UniqueName: \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-kube-api-access-td47r\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.164740    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.164781    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.164862    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.164887    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.164939    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.165189    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e0b9a5-206c-4589-a9dc-4b4f46620605-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.165325    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.165489    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.166038    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-credential-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.168027    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-socket\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.168714    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e0b9a5-206c-4589-a9dc-4b4f46620605-istiod-ca-cert\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.171394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.169011    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-certs\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.185405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.173394    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-envoy\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.185405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.173797    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-data\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.191530 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.186716    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td47r\" (UniqueName: \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-kube-api-access-td47r\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.191530 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.189436    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:38.191530 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.190166    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-token\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.206754 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.192962    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-podinfo\") pod \"e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") " pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.451553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.448725    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:38.552706 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:52:38.552671    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4c30d5_2321_45c8_8a25_ce66a86629a6.slice/crio-def039d0c9fd51952117b0a02cda2bc833dff145444942ac5199f442a757c0c9 WatchSource:0}: Error finding container def039d0c9fd51952117b0a02cda2bc833dff145444942ac5199f442a757c0c9: Status 404 returned error can't find the container with id def039d0c9fd51952117b0a02cda2bc833dff145444942ac5199f442a757c0c9
Oct 30 23:52:38.563055 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.562989    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:38.563185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.563156    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:38.563312 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.563290    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:38.584619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.583530    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"]
Oct 30 23:52:38.853447 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:52:38.853364    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e0b9a5_206c_4589_a9dc_4b4f46620605.slice/crio-51541d753b44105d7e53a7c6009b68ba026859393309f599cdd94d0c3799b59d WatchSource:0}: Error finding container 51541d753b44105d7e53a7c6009b68ba026859393309f599cdd94d0c3799b59d: Status 404 returned error can't find the container with id 51541d753b44105d7e53a7c6009b68ba026859393309f599cdd94d0c3799b59d
Oct 30 23:52:38.855183 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.855159    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"]
Oct 30 23:52:38.856884 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.856851    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:38.856969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.856921    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:38.856969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:38.856950    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:52:39.069165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.069129    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" event={"ID":"77e0b9a5-206c-4589-a9dc-4b4f46620605","Type":"ContainerStarted","Data":"3c135d038be4c16cba2ebf437fb8115bbc791e7d6bbe24872ad729168513ca30"}
Oct 30 23:52:39.069348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.069191    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" event={"ID":"77e0b9a5-206c-4589-a9dc-4b4f46620605","Type":"ContainerStarted","Data":"51541d753b44105d7e53a7c6009b68ba026859393309f599cdd94d0c3799b59d"}
Oct 30 23:52:39.072320 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.072277    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" event={"ID":"9d4c30d5-2321-45c8-8a25-ce66a86629a6","Type":"ContainerStarted","Data":"5073d300c5d7359d181fcbb6151596c94e1f7e95cebf812c49ba23068b860944"}
Oct 30 23:52:39.072514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.072340    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" event={"ID":"9d4c30d5-2321-45c8-8a25-ce66a86629a6","Type":"ContainerStarted","Data":"def039d0c9fd51952117b0a02cda2bc833dff145444942ac5199f442a757c0c9"}
Oct 30 23:52:39.101014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.100897    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" podStartSLOduration=2.100875919 podStartE2EDuration="2.100875919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:39.093421284 +0000 UTC m=+2018.020058867" watchObservedRunningTime="2026-10-30 23:52:39.100875919 +0000 UTC m=+2018.027513504"
Oct 30 23:52:39.123174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.123106    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" podStartSLOduration=2.1230838 podStartE2EDuration="2.1230838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:39.121631605 +0000 UTC m=+2018.048269185" watchObservedRunningTime="2026-10-30 23:52:39.1230838 +0000 UTC m=+2018.049721385"
Oct 30 23:52:39.190596 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.190560    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:39.191997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.191968    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 23:52:39.192124 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.192030    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 23:52:39.449648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.449548    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:39.450741 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.450647    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 23:52:39.450741 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:39.450700    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 23:52:40.201662 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:40.191765    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 23:52:40.201662 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:40.191827    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 23:52:40.460445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:40.457847    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 23:52:40.460445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:40.457905    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 23:52:41.190740 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:41.190702    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 23:52:41.192357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:41.192322    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 23:52:41.449684 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:41.449580    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 23:52:41.450126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:41.450102    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 23:52:42.192403 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:42.191744    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 23:52:42.192741 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:42.192703    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 23:52:42.450001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:42.449917    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 23:52:42.450001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:42.449975    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 23:52:43.192397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:43.191701    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 23:52:43.192397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:43.191770    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 23:52:43.450081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:43.449991    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 23:52:43.450081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:43.450071    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 23:52:44.202395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:44.201857    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 23:52:44.202395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:44.201914    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 23:52:44.449859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:44.449826    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 23:52:44.450034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:44.449877    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 23:52:45.193395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:45.193341    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 23:52:45.193942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:45.193425    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 23:52:45.449564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:45.449484    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 23:52:45.449564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:45.449539    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 23:52:46.196644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:46.196612    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:46.197111 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:46.196948    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:46.197846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:46.197823    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:52:46.455200 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:46.455119    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:46.455664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:46.455643    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:46.456327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:46.456308    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:52:52.032434 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.032393    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"]
Oct 30 23:52:52.038778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.038746    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d67dc42-lwttt"]
Oct 30 23:52:52.077992 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.077954    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"]
Oct 30 23:52:52.084106 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.084079    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.089716 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.089675    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:52:52.090927 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.090891    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:52:52.092683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.092659    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"]
Oct 30 23:52:52.113483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.112903    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8kjh\" (UniqueName: \"kubernetes.io/projected/774a17c8-1193-431c-9319-8e30de37ead9-kube-api-access-p8kjh\") pod \"maas-api-revoke-sub-08ce4ee8-8cs74\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.113483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.112979    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ce4ee8-8cs74\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.113483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.113199    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ce4ee8-8cs74\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.214997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.214952    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8kjh\" (UniqueName: \"kubernetes.io/projected/774a17c8-1193-431c-9319-8e30de37ead9-kube-api-access-p8kjh\") pod \"maas-api-revoke-sub-08ce4ee8-8cs74\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.214997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.215005    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ce4ee8-8cs74\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.215263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.215114    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ce4ee8-8cs74\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.215890 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.215863    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ce4ee8-8cs74\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.217634 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.217614    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ce4ee8-8cs74\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.226655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.226632    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8kjh\" (UniqueName: \"kubernetes.io/projected/774a17c8-1193-431c-9319-8e30de37ead9-kube-api-access-p8kjh\") pod \"maas-api-revoke-sub-08ce4ee8-8cs74\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.418319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.418230    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:52.723573 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:52:52.723532    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774a17c8_1193_431c_9319_8e30de37ead9.slice/crio-9711a186c7fb0770bd1e11cc1e6aac5ec304f7f94ec40ab9ba40dcc3919d50f0 WatchSource:0}: Error finding container 9711a186c7fb0770bd1e11cc1e6aac5ec304f7f94ec40ab9ba40dcc3919d50f0: Status 404 returned error can't find the container with id 9711a186c7fb0770bd1e11cc1e6aac5ec304f7f94ec40ab9ba40dcc3919d50f0
Oct 30 23:52:52.724044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:52.724022    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"]
Oct 30 23:52:53.212639 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:53.212602    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7072f6a6-53fe-45cc-bae1-35637abda998" path="/var/lib/kubelet/pods/7072f6a6-53fe-45cc-bae1-35637abda998/volumes"
Oct 30 23:52:53.345125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:53.345088    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74" event={"ID":"774a17c8-1193-431c-9319-8e30de37ead9","Type":"ContainerStarted","Data":"931bf0030239938932709164c65d112c64cbf40729d145776a63ff4b7ce459a5"}
Oct 30 23:52:53.345317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:53.345139    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74" event={"ID":"774a17c8-1193-431c-9319-8e30de37ead9","Type":"ContainerStarted","Data":"9711a186c7fb0770bd1e11cc1e6aac5ec304f7f94ec40ab9ba40dcc3919d50f0"}
Oct 30 23:52:53.366991 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:53.366938    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74" podStartSLOduration=1.366920567 podStartE2EDuration="1.366920567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:53.364049861 +0000 UTC m=+2032.290687442" watchObservedRunningTime="2026-10-30 23:52:53.366920567 +0000 UTC m=+2032.293558147"
Oct 30 23:52:54.355999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:54.355962    4555 generic.go:356] "Generic (PLEG): container finished" podID="774a17c8-1193-431c-9319-8e30de37ead9" containerID="931bf0030239938932709164c65d112c64cbf40729d145776a63ff4b7ce459a5" exitCode=0
Oct 30 23:52:54.356496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:54.356109    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74" event={"ID":"774a17c8-1193-431c-9319-8e30de37ead9","Type":"ContainerDied","Data":"931bf0030239938932709164c65d112c64cbf40729d145776a63ff4b7ce459a5"}
Oct 30 23:52:55.571541 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:55.571510    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:55.681010 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:55.680785    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-openshift-ca\") pod \"774a17c8-1193-431c-9319-8e30de37ead9\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") "
Oct 30 23:52:55.681010 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:55.681018    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774a17c8-1193-431c-9319-8e30de37ead9-kube-api-access-p8kjh\" (UniqueName: \"kubernetes.io/projected/774a17c8-1193-431c-9319-8e30de37ead9-kube-api-access-p8kjh\") pod \"774a17c8-1193-431c-9319-8e30de37ead9\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") "
Oct 30 23:52:55.681315 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:55.681287    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-openshift-ca" pod "774a17c8-1193-431c-9319-8e30de37ead9" (UID: "774a17c8-1193-431c-9319-8e30de37ead9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:52:55.681491 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:55.681471    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-xks-ca\") pod \"774a17c8-1193-431c-9319-8e30de37ead9\" (UID: \"774a17c8-1193-431c-9319-8e30de37ead9\") "
Oct 30 23:52:55.682625 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:55.682596    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:52:55.684219 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:55.684177    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774a17c8-1193-431c-9319-8e30de37ead9-kube-api-access-p8kjh" pod "774a17c8-1193-431c-9319-8e30de37ead9" (UID: "774a17c8-1193-431c-9319-8e30de37ead9"). InnerVolumeSpecName "kube-api-access-p8kjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:52:55.684477 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:55.684455    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-xks-ca" pod "774a17c8-1193-431c-9319-8e30de37ead9" (UID: "774a17c8-1193-431c-9319-8e30de37ead9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:52:55.784318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:55.784278    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8kjh\" (UniqueName: \"kubernetes.io/projected/774a17c8-1193-431c-9319-8e30de37ead9-kube-api-access-p8kjh\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:52:55.784318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:55.784319    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774a17c8-1193-431c-9319-8e30de37ead9-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:52:56.395702 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:56.395656    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74" event={"ID":"774a17c8-1193-431c-9319-8e30de37ead9","Type":"ContainerDied","Data":"9711a186c7fb0770bd1e11cc1e6aac5ec304f7f94ec40ab9ba40dcc3919d50f0"}
Oct 30 23:52:56.395955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:56.395754    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9711a186c7fb0770bd1e11cc1e6aac5ec304f7f94ec40ab9ba40dcc3919d50f0"
Oct 30 23:52:56.395955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:52:56.395681    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"
Oct 30 23:52:57.447867 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:52:57.447826    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:53:02.997928 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:02.997896    4555 scope.go:122] "RemoveContainer" containerID="81b9605695f620c26d3a2f00ab163c60609999e26e5c32c147aafc76412fce31"
Oct 30 23:53:03.010888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:03.010864    4555 scope.go:122] "RemoveContainer" containerID="3323d1d0afc2d16db881bd3f4766a294d909dc27d9b6119cb0c7f5f84ff28487"
Oct 30 23:53:03.022465 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:03.022445    4555 scope.go:122] "RemoveContainer" containerID="904e111112002dbfad3860ba5cad388d15bac08217ff1c6cf4cce096d99911ed"
Oct 30 23:53:27.448890 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:53:27.448847    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:53:29.836396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:29.832466    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt" podUID="f832f680-ca9d-434b-ac35-3c9c47105f8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.157:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:29.854475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:29.852501    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v" podUID="648f4553-fa26-444b-a2e4-a690b559990f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:29.912441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:29.910464    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm" podUID="0cbd6143-2e09-46e2-b701-f1e457c32d1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:29.957394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:29.956508    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.152:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.177448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.176541    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.211394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.211332    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l" podUID="68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.300397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.294456    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.152:8000/health\": context deadline exceeded"
Oct 30 23:53:30.412396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.411466    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.462454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.459477    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Oct 30 23:53:30.511396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.504475    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.531394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.526454    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.531608 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.531492    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.602398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.598586    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": context deadline exceeded"
Oct 30 23:53:30.606650 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.606617    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:53:30.607158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.607127    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.654988 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.654945    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.699339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.699298    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl" podUID="0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.156:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.762527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.762485    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.835544 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.835499    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt" podUID="f832f680-ca9d-434b-ac35-3c9c47105f8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.157:8000/ready\": context deadline exceeded"
Oct 30 23:53:30.836241 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.836217    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:53:30.853510 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.853475    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v" podUID="648f4553-fa26-444b-a2e4-a690b559990f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.854674 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.854652    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:53:30.936395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.933046    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm" podUID="0cbd6143-2e09-46e2-b701-f1e457c32d1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.969636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.969604    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:53:30.995406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.970960    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.152:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:30.997016 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:30.996976    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:53:31.249661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.247846    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-mjk4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:53:31.249661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.247929    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" podUID="ea819a48-154f-4c7a-a747-3306ac423289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:31.249661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.226610    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l" podUID="68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": context deadline exceeded"
Oct 30 23:53:31.249661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.226551    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l" podUID="68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/health\": context deadline exceeded"
Oct 30 23:53:31.249661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.201798    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:31.363401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.352713    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:31.413098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.412262    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:31.477389 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.477312    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Oct 30 23:53:31.498545 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.497442    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm" podUID="0cbd6143-2e09-46e2-b701-f1e457c32d1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:31.575315 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.575272    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:31.591831 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.591800    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:53:31.592153 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.592122    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:53:31.592322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.592309    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:53:31.592474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.592460    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:53:31.592636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.592623    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:53:31.617149 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.615551    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 23:53:31.617149 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.615635    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": context deadline exceeded"
Oct 30 23:53:31.648968 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.644117    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:53:31.648968 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.644188    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:53:31.648968 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.644226    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:53:31.651993 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.651886    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:53:31.684393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.617074    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:31.684393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.667880    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:53:31.711797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.690547    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:31.728628 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.728583    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl" podUID="0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:53:31.751304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.751278    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:53:31.807813 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.807784    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"]
Oct 30 23:53:31.808148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.808130    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:53:31.832606 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.832525    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:53:31.832856 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:31.832828    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:53:32.025555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:32.025519    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e131ad43-ck7sl"]
Oct 30 23:53:32.113313 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:32.113231    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:53:32.124611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:32.124581    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 30 23:53:32.124836 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:32.124826    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:53:32.125033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:32.125014    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 30 23:53:32.125186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:32.125175    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:53:32.125336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:32.125326    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:53:32.125511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:32.125501    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 30 23:53:32.144389 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:32.142595    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:53:33.212858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:53:33.212826    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec03e339-604b-4ef8-8646-957af49d386e" path="/var/lib/kubelet/pods/ec03e339-604b-4ef8-8646-957af49d386e/volumes"
Oct 30 23:53:57.449751 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:53:57.449711    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:54:03.078307 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:03.078268    4555 scope.go:122] "RemoveContainer" containerID="7f4c076c12d5f30f0f8273a7f9582ef01f3cb925a8805b959a8e9ebb0293e97f"
Oct 30 23:54:03.095151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:03.090970    4555 scope.go:122] "RemoveContainer" containerID="e964fb2ba9299f6bce3f1c91b6838377ddf6c1f2eb4209dd4667661c159f8596"
Oct 30 23:54:03.110150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:03.110128    4555 scope.go:122] "RemoveContainer" containerID="ba1f1d763d65c65c47f8fa5321ba9e85f40158b37c8e984bd43e2a2b87611252"
Oct 30 23:54:16.590271 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.590073    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7845b665bc-52x8q"]
Oct 30 23:54:16.592063 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.592037    4555 cpu_manager.go:414] "containerMap: removing container" podUID="774a17c8-1193-431c-9319-8e30de37ead9" containerName="revoke-keys"
Oct 30 23:54:16.592227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.592211    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774a17c8-1193-431c-9319-8e30de37ead9" containerName="revoke-keys"
Oct 30 23:54:16.592623 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.592607    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="774a17c8-1193-431c-9319-8e30de37ead9" containerName="revoke-keys"
Oct 30 23:54:16.600344 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.599583    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.622063 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.622034    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7845b665bc-52x8q"]
Oct 30 23:54:16.745885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.745832    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6b71183-9683-421d-84bd-bb1225ada1a9-tmp\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.746083 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.745891    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-console-config\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.746083 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.746004    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-trusted-ca-bundle\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.746216 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.746174    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-service-ca\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.746283 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.746224    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxhm\" (UniqueName: \"kubernetes.io/projected/a6b71183-9683-421d-84bd-bb1225ada1a9-kube-api-access-plxhm\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.746342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.746299    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-oauth-serving-cert\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.746419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.746404    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-oauth-config\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.746897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.746624    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-serving-cert\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.863019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.859184    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-oauth-config\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.863019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.859331    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-serving-cert\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.863019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.859533    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6b71183-9683-421d-84bd-bb1225ada1a9-tmp\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.863019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.859566    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-console-config\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.863019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.859609    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-trusted-ca-bundle\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.879542 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.859765    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-service-ca\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.879814 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.879793    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxhm\" (UniqueName: \"kubernetes.io/projected/a6b71183-9683-421d-84bd-bb1225ada1a9-kube-api-access-plxhm\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.880319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.880298    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-oauth-serving-cert\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.880548 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.860622    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-service-ca\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.880846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.874885    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-oauth-config\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.880957 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.879506    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-trusted-ca-bundle\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.881067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.860588    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6b71183-9683-421d-84bd-bb1225ada1a9-tmp\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.881158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.860614    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-console-config\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.895390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.882448    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-oauth-serving-cert\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.908413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.905588    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-serving-cert\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.917447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.910615    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxhm\" (UniqueName: \"kubernetes.io/projected/a6b71183-9683-421d-84bd-bb1225ada1a9-kube-api-access-plxhm\") pod \"console-7845b665bc-52x8q\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") " pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:16.938275 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:16.938241    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:17.291961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:17.291623    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7845b665bc-52x8q"]
Oct 30 23:54:17.296843 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:54:17.296270    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b71183_9683_421d_84bd_bb1225ada1a9.slice/crio-8fd409414c03671798741ffde74e2cade25291f94eafe8cfb58e94187932404b WatchSource:0}: Error finding container 8fd409414c03671798741ffde74e2cade25291f94eafe8cfb58e94187932404b: Status 404 returned error can't find the container with id 8fd409414c03671798741ffde74e2cade25291f94eafe8cfb58e94187932404b
Oct 30 23:54:18.293946 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:18.292580    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7845b665bc-52x8q" event={"ID":"a6b71183-9683-421d-84bd-bb1225ada1a9","Type":"ContainerStarted","Data":"9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e"}
Oct 30 23:54:18.293946 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:18.292647    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7845b665bc-52x8q" event={"ID":"a6b71183-9683-421d-84bd-bb1225ada1a9","Type":"ContainerStarted","Data":"8fd409414c03671798741ffde74e2cade25291f94eafe8cfb58e94187932404b"}
Oct 30 23:54:20.067145 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.067068    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7845b665bc-52x8q" podStartSLOduration=4.067048256 podStartE2EDuration="4.067048256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:18.32780381 +0000 UTC m=+2117.254441392" watchObservedRunningTime="2026-10-30 23:54:20.067048256 +0000 UTC m=+2118.993685841"
Oct 30 23:54:20.079907 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.079866    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"]
Oct 30 23:54:20.085990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.085964    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"]
Oct 30 23:54:20.086164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.086071    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.097416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.096193    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:54:20.097416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.096306    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:54:20.195687 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.195651    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72996a61-lz7vk\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.196097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.196074    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72996a61-lz7vk\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.196188 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.196127    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b85mk\" (UniqueName: \"kubernetes.io/projected/bee07856-194a-4473-8e18-831ed49bc93a-kube-api-access-b85mk\") pod \"maas-api-revoke-sub-72996a61-lz7vk\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.298530 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.298489    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72996a61-lz7vk\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.298758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.298548    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b85mk\" (UniqueName: \"kubernetes.io/projected/bee07856-194a-4473-8e18-831ed49bc93a-kube-api-access-b85mk\") pod \"maas-api-revoke-sub-72996a61-lz7vk\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.298758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.298630    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72996a61-lz7vk\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.299589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.299560    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72996a61-lz7vk\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.301350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.301323    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72996a61-lz7vk\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.314681 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.314651    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b85mk\" (UniqueName: \"kubernetes.io/projected/bee07856-194a-4473-8e18-831ed49bc93a-kube-api-access-b85mk\") pod \"maas-api-revoke-sub-72996a61-lz7vk\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.430790 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.430707    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:20.731913 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:54:20.731884    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee07856_194a_4473_8e18_831ed49bc93a.slice/crio-fece2cfe0a1eda1aea663783f7632bcbb74d767091bac32de616de778a0a697b WatchSource:0}: Error finding container fece2cfe0a1eda1aea663783f7632bcbb74d767091bac32de616de778a0a697b: Status 404 returned error can't find the container with id fece2cfe0a1eda1aea663783f7632bcbb74d767091bac32de616de778a0a697b
Oct 30 23:54:20.732997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:20.732964    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"]
Oct 30 23:54:21.325667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:21.325634    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk" event={"ID":"bee07856-194a-4473-8e18-831ed49bc93a","Type":"ContainerStarted","Data":"8928a1a4fb3b6ace4c20f81500bc72868777ececbcdad0cda5e33968c02ae294"}
Oct 30 23:54:21.326719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:21.325685    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk" event={"ID":"bee07856-194a-4473-8e18-831ed49bc93a","Type":"ContainerStarted","Data":"fece2cfe0a1eda1aea663783f7632bcbb74d767091bac32de616de778a0a697b"}
Oct 30 23:54:21.351179 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:21.351124    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk" podStartSLOduration=1.351107008 podStartE2EDuration="1.351107008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:21.349625075 +0000 UTC m=+2120.276262655" watchObservedRunningTime="2026-10-30 23:54:21.351107008 +0000 UTC m=+2120.277744590"
Oct 30 23:54:22.336348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:22.336310    4555 generic.go:356] "Generic (PLEG): container finished" podID="bee07856-194a-4473-8e18-831ed49bc93a" containerID="8928a1a4fb3b6ace4c20f81500bc72868777ececbcdad0cda5e33968c02ae294" exitCode=0
Oct 30 23:54:22.336797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:22.336394    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk" event={"ID":"bee07856-194a-4473-8e18-831ed49bc93a","Type":"ContainerDied","Data":"8928a1a4fb3b6ace4c20f81500bc72868777ececbcdad0cda5e33968c02ae294"}
Oct 30 23:54:22.509856 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:22.509823    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:54:23.585592 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:23.585565    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:23.720930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:23.720842    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee07856-194a-4473-8e18-831ed49bc93a-kube-api-access-b85mk\" (UniqueName: \"kubernetes.io/projected/bee07856-194a-4473-8e18-831ed49bc93a-kube-api-access-b85mk\") pod \"bee07856-194a-4473-8e18-831ed49bc93a\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") "
Oct 30 23:54:23.720930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:23.720926    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-xks-ca\") pod \"bee07856-194a-4473-8e18-831ed49bc93a\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") "
Oct 30 23:54:23.721191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:23.720952    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-openshift-ca\") pod \"bee07856-194a-4473-8e18-831ed49bc93a\" (UID: \"bee07856-194a-4473-8e18-831ed49bc93a\") "
Oct 30 23:54:23.721449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:23.721423    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-openshift-ca" pod "bee07856-194a-4473-8e18-831ed49bc93a" (UID: "bee07856-194a-4473-8e18-831ed49bc93a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:23.721844 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:23.721824    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:23.723317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:23.723291    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-xks-ca" pod "bee07856-194a-4473-8e18-831ed49bc93a" (UID: "bee07856-194a-4473-8e18-831ed49bc93a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:23.723446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:23.723418    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee07856-194a-4473-8e18-831ed49bc93a-kube-api-access-b85mk" pod "bee07856-194a-4473-8e18-831ed49bc93a" (UID: "bee07856-194a-4473-8e18-831ed49bc93a"). InnerVolumeSpecName "kube-api-access-b85mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:23.824296 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:23.824257    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee07856-194a-4473-8e18-831ed49bc93a-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:23.824296 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:23.824296    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b85mk\" (UniqueName: \"kubernetes.io/projected/bee07856-194a-4473-8e18-831ed49bc93a-kube-api-access-b85mk\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:24.494401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:24.476183    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk" event={"ID":"bee07856-194a-4473-8e18-831ed49bc93a","Type":"ContainerDied","Data":"fece2cfe0a1eda1aea663783f7632bcbb74d767091bac32de616de778a0a697b"}
Oct 30 23:54:24.494770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:24.494741    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fece2cfe0a1eda1aea663783f7632bcbb74d767091bac32de616de778a0a697b"
Oct 30 23:54:24.494924 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:24.485489    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"
Oct 30 23:54:26.939517 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:26.939481    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:26.940143 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:26.940125    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:26.957314 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:26.956472    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:27.452203 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:54:27.452161    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:54:27.510610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:27.510576    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:54:27.574571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:27.574535    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b99cfb955-4zgzj"]
Oct 30 23:54:52.607034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:52.606991    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7b99cfb955-4zgzj" podUID="c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" containerName="console" containerID="cri-o://5d6c8ae79249c9d22eb75477e9b226393d0033864339d89de744aa16b24a4b7f" gracePeriod=15
Oct 30 23:54:52.858563 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:52.858478    4555 generic.go:356] "Generic (PLEG): container finished" podID="c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" containerID="5d6c8ae79249c9d22eb75477e9b226393d0033864339d89de744aa16b24a4b7f" exitCode=0
Oct 30 23:54:52.858563 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:52.858548    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b99cfb955-4zgzj" event={"ID":"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b","Type":"ContainerDied","Data":"5d6c8ae79249c9d22eb75477e9b226393d0033864339d89de744aa16b24a4b7f"}
Oct 30 23:54:53.013531 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.013502    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:54:53.139291 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.139159    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-serving-cert\") pod \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") "
Oct 30 23:54:53.139498 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.139317    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-trusted-ca-bundle\") pod \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") "
Oct 30 23:54:53.139586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.139508    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-service-ca\") pod \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") "
Oct 30 23:54:53.139586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.139576    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-kube-api-access-q9d87\" (UniqueName: \"kubernetes.io/projected/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-kube-api-access-q9d87\") pod \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") "
Oct 30 23:54:53.139732 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.139702    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-trusted-ca-bundle" pod "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" (UID: "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:53.139808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.139758    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-tmp\") pod \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") "
Oct 30 23:54:53.139924 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.139898    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-service-ca" pod "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" (UID: "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:53.139924 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.139917    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-config\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-config\") pod \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") "
Oct 30 23:54:53.140099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.140007    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-tmp" pod "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" (UID: "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:53.140099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.140046    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-oauth-config\") pod \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") "
Oct 30 23:54:53.140220 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.140145    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-oauth-serving-cert\") pod \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\" (UID: \"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b\") "
Oct 30 23:54:53.140468 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.140440    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-config" pod "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" (UID: "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:53.140597 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.140573    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-oauth-serving-cert" pod "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" (UID: "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:53.141203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.141180    4555 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-tmp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.141310 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.141207    4555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-config\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.141310 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.141223    4555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-oauth-serving-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.141310 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.141238    4555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-trusted-ca-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.141310 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.141252    4555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.142065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.142041    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-serving-cert" pod "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" (UID: "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:53.142430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.142399    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-kube-api-access-q9d87" pod "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" (UID: "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b"). InnerVolumeSpecName "kube-api-access-q9d87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:53.142430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.142418    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-oauth-config" pod "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" (UID: "c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:53.243693 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.243664    4555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-serving-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.243693 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.243695    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9d87\" (UniqueName: \"kubernetes.io/projected/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-kube-api-access-q9d87\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.243884 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.243709    4555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b-console-oauth-config\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.870783 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.870751    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b99cfb955-4zgzj"
Oct 30 23:54:53.870783 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.870773    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b99cfb955-4zgzj" event={"ID":"c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b","Type":"ContainerDied","Data":"b8b98bd99caa8a5f71edb157b4506410a1ecc6de72b6317a60c5c199e9bd9bd4"}
Oct 30 23:54:53.871437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.870845    4555 scope.go:122] "RemoveContainer" containerID="5d6c8ae79249c9d22eb75477e9b226393d0033864339d89de744aa16b24a4b7f"
Oct 30 23:54:53.898632 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.898604    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b99cfb955-4zgzj"]
Oct 30 23:54:53.906794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:53.906767    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b99cfb955-4zgzj"]
Oct 30 23:54:54.245828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:54.245753    4555 ???:1] "http: TLS handshake error from 10.0.137.81:33421: read tcp 10.0.134.83:10250->10.0.137.81:33421: read: connection reset by peer"
Oct 30 23:54:55.212933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:54:55.212901    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" path="/var/lib/kubelet/pods/c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b/volumes"
Oct 30 23:54:57.448350 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:54:57.448313    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:55:03.157505 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:03.157472    4555 scope.go:122] "RemoveContainer" containerID="c592548c6a72aa7e905ebd605aea3b144d76ee84df4aa46f37ac03948325c712"
Oct 30 23:55:03.171980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:03.171951    4555 scope.go:122] "RemoveContainer" containerID="f4d4312b7e0c50e45e904ae079aeb55ad9e9cb78d5f12664e44c9a93b9c74fbd"
Oct 30 23:55:27.307934 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.307900    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"]
Oct 30 23:55:27.308759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.308741    4555 cpu_manager.go:414] "containerMap: removing container" podUID="c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" containerName="console"
Oct 30 23:55:27.308817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.308761    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" containerName="console"
Oct 30 23:55:27.308817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.308807    4555 cpu_manager.go:414] "containerMap: removing container" podUID="bee07856-194a-4473-8e18-831ed49bc93a" containerName="revoke-keys"
Oct 30 23:55:27.308817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.308813    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee07856-194a-4473-8e18-831ed49bc93a" containerName="revoke-keys"
Oct 30 23:55:27.308943 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.308934    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee07856-194a-4473-8e18-831ed49bc93a" containerName="revoke-keys"
Oct 30 23:55:27.308989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.308951    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8aa6bf5-2dc1-4b1f-97f3-78301e16fa5b" containerName="console"
Oct 30 23:55:27.312779 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.312756    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:27.319432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.317575    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:55:27.319432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.318867    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:55:27.328762 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.325809    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"]
Oct 30 23:55:27.450684 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:55:27.450642    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:55:27.489832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.489798    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:27.490066 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.490043    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49tk\" (UniqueName: \"kubernetes.io/projected/11457ba3-82a3-4994-a1ab-29ffb4046b04-kube-api-access-m49tk\") pod \"maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:27.490185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.490166    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:27.592123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.592033    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49tk\" (UniqueName: \"kubernetes.io/projected/11457ba3-82a3-4994-a1ab-29ffb4046b04-kube-api-access-m49tk\") pod \"maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:27.592303 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.592124    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:27.592395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.592320    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:27.593002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.592964    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:27.595064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.595041    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:27.604940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.604904    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49tk\" (UniqueName: \"kubernetes.io/projected/11457ba3-82a3-4994-a1ab-29ffb4046b04-kube-api-access-m49tk\") pod \"maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:27.647757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:27.647722    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:28.006567 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:55:28.006534    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11457ba3_82a3_4994_a1ab_29ffb4046b04.slice/crio-7642633fa7d0c0a9e1e1ae30d94631f08615815a94b1cba2a3847adde1a8f643 WatchSource:0}: Error finding container 7642633fa7d0c0a9e1e1ae30d94631f08615815a94b1cba2a3847adde1a8f643: Status 404 returned error can't find the container with id 7642633fa7d0c0a9e1e1ae30d94631f08615815a94b1cba2a3847adde1a8f643
Oct 30 23:55:28.007057 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:28.007033    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"]
Oct 30 23:55:28.381097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:28.381010    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx" event={"ID":"11457ba3-82a3-4994-a1ab-29ffb4046b04","Type":"ContainerStarted","Data":"7642633fa7d0c0a9e1e1ae30d94631f08615815a94b1cba2a3847adde1a8f643"}
Oct 30 23:55:29.397249 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:29.397153    4555 generic.go:356] "Generic (PLEG): container finished" podID="11457ba3-82a3-4994-a1ab-29ffb4046b04" containerID="2770ff4b2d3daafe20a43ff4ea08e170c829741f91ed55ec5a04d3c186241ebb" exitCode=0
Oct 30 23:55:29.397799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:29.397445    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx" event={"ID":"11457ba3-82a3-4994-a1ab-29ffb4046b04","Type":"ContainerDied","Data":"2770ff4b2d3daafe20a43ff4ea08e170c829741f91ed55ec5a04d3c186241ebb"}
Oct 30 23:55:30.522767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.522731    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"]
Oct 30 23:55:30.527970 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.527942    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.532834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.532779    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"]
Oct 30 23:55:30.594537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.594502    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqn9\" (UniqueName: \"kubernetes.io/projected/767dcd83-f5fd-46a7-ad3c-73361d87316e-kube-api-access-8lqn9\") pod \"maas-api-revoke-sub-91354e50-sqrn9\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.594969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.594950    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-sqrn9\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.595165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.595147    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-sqrn9\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.696865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.696832    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-sqrn9\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.697034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.696890    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-sqrn9\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.697288 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.697270    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqn9\" (UniqueName: \"kubernetes.io/projected/767dcd83-f5fd-46a7-ad3c-73361d87316e-kube-api-access-8lqn9\") pod \"maas-api-revoke-sub-91354e50-sqrn9\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.697585 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.697561    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-sqrn9\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.699885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.699863    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-sqrn9\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.709724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.709700    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqn9\" (UniqueName: \"kubernetes.io/projected/767dcd83-f5fd-46a7-ad3c-73361d87316e-kube-api-access-8lqn9\") pod \"maas-api-revoke-sub-91354e50-sqrn9\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.741084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.741056    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:30.798483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.798449    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11457ba3-82a3-4994-a1ab-29ffb4046b04-kube-api-access-m49tk\" (UniqueName: \"kubernetes.io/projected/11457ba3-82a3-4994-a1ab-29ffb4046b04-kube-api-access-m49tk\") pod \"11457ba3-82a3-4994-a1ab-29ffb4046b04\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") "
Oct 30 23:55:30.798747 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.798730    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-xks-ca\") pod \"11457ba3-82a3-4994-a1ab-29ffb4046b04\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") "
Oct 30 23:55:30.798880 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.798852    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-openshift-ca\") pod \"11457ba3-82a3-4994-a1ab-29ffb4046b04\" (UID: \"11457ba3-82a3-4994-a1ab-29ffb4046b04\") "
Oct 30 23:55:30.799308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.799280    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-openshift-ca" pod "11457ba3-82a3-4994-a1ab-29ffb4046b04" (UID: "11457ba3-82a3-4994-a1ab-29ffb4046b04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:55:30.800128 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.799835    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:30.803578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.803462    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-xks-ca" pod "11457ba3-82a3-4994-a1ab-29ffb4046b04" (UID: "11457ba3-82a3-4994-a1ab-29ffb4046b04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:55:30.803578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.803540    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11457ba3-82a3-4994-a1ab-29ffb4046b04-kube-api-access-m49tk" pod "11457ba3-82a3-4994-a1ab-29ffb4046b04" (UID: "11457ba3-82a3-4994-a1ab-29ffb4046b04"). InnerVolumeSpecName "kube-api-access-m49tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:55:30.855173 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.855144    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:30.901973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.901939    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m49tk\" (UniqueName: \"kubernetes.io/projected/11457ba3-82a3-4994-a1ab-29ffb4046b04-kube-api-access-m49tk\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:30.901973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:30.901973    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11457ba3-82a3-4994-a1ab-29ffb4046b04-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:31.207347 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:31.206188    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"]
Oct 30 23:55:31.417659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:31.417621    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9" event={"ID":"767dcd83-f5fd-46a7-ad3c-73361d87316e","Type":"ContainerStarted","Data":"7ea2f045088640be943174eeb99833294093e4f2191ec676aecb3ab4958f4c23"}
Oct 30 23:55:31.419168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:31.419143    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx" event={"ID":"11457ba3-82a3-4994-a1ab-29ffb4046b04","Type":"ContainerDied","Data":"7642633fa7d0c0a9e1e1ae30d94631f08615815a94b1cba2a3847adde1a8f643"}
Oct 30 23:55:31.419313 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:31.419186    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7642633fa7d0c0a9e1e1ae30d94631f08615815a94b1cba2a3847adde1a8f643"
Oct 30 23:55:31.419313 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:31.419152    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"
Oct 30 23:55:32.429698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:32.429613    4555 generic.go:356] "Generic (PLEG): container finished" podID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerID="e91e5574a588c62685f2320ce3e0aad51a4a1300be152483d7b984a8b36fd445" exitCode=7
Oct 30 23:55:32.430079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:32.429699    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9" event={"ID":"767dcd83-f5fd-46a7-ad3c-73361d87316e","Type":"ContainerDied","Data":"e91e5574a588c62685f2320ce3e0aad51a4a1300be152483d7b984a8b36fd445"}
Oct 30 23:55:32.430079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:32.430072    4555 scope.go:122] "RemoveContainer" containerID="e91e5574a588c62685f2320ce3e0aad51a4a1300be152483d7b984a8b36fd445"
Oct 30 23:55:33.440964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:33.440930    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9" event={"ID":"767dcd83-f5fd-46a7-ad3c-73361d87316e","Type":"ContainerStarted","Data":"302a52c0f3004db2f796ee96b4f15e257cd845ec298a0bf426dcad19ef59dd5b"}
Oct 30 23:55:33.485163 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:33.485102    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9" podStartSLOduration=3.485083162 podStartE2EDuration="3.485083162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:33.477837329 +0000 UTC m=+2192.404474911" watchObservedRunningTime="2026-10-30 23:55:33.485083162 +0000 UTC m=+2192.411720744"
Oct 30 23:55:34.456473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:34.456433    4555 generic.go:356] "Generic (PLEG): container finished" podID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerID="302a52c0f3004db2f796ee96b4f15e257cd845ec298a0bf426dcad19ef59dd5b" exitCode=7
Oct 30 23:55:34.457001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:34.456484    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9" event={"ID":"767dcd83-f5fd-46a7-ad3c-73361d87316e","Type":"ContainerDied","Data":"302a52c0f3004db2f796ee96b4f15e257cd845ec298a0bf426dcad19ef59dd5b"}
Oct 30 23:55:34.457001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:34.456539    4555 scope.go:122] "RemoveContainer" containerID="e91e5574a588c62685f2320ce3e0aad51a4a1300be152483d7b984a8b36fd445"
Oct 30 23:55:34.457175 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:34.457155    4555 scope.go:122] "RemoveContainer" containerID="302a52c0f3004db2f796ee96b4f15e257cd845ec298a0bf426dcad19ef59dd5b"
Oct 30 23:55:34.457915 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:55:34.457717    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-sqrn9_odh-ai-gateway-infra(767dcd83-f5fd-46a7-ad3c-73361d87316e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e"
Oct 30 23:55:35.467801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:35.467769    4555 scope.go:122] "RemoveContainer" containerID="302a52c0f3004db2f796ee96b4f15e257cd845ec298a0bf426dcad19ef59dd5b"
Oct 30 23:55:35.468318 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:55:35.468190    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-sqrn9_odh-ai-gateway-infra(767dcd83-f5fd-46a7-ad3c-73361d87316e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e"
Oct 30 23:55:44.616631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.616552    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"]
Oct 30 23:55:44.617341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.617326    4555 cpu_manager.go:414] "containerMap: removing container" podUID="11457ba3-82a3-4994-a1ab-29ffb4046b04" containerName="revoke-keys"
Oct 30 23:55:44.617417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.617343    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11457ba3-82a3-4994-a1ab-29ffb4046b04" containerName="revoke-keys"
Oct 30 23:55:44.617564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.617550    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="11457ba3-82a3-4994-a1ab-29ffb4046b04" containerName="revoke-keys"
Oct 30 23:55:44.622791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.622773    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:44.643823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.643791    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"]
Oct 30 23:55:44.757290 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.757240    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b13a079-9236-4642-b193-77b4903352c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:44.757680 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.757349    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqtcf\" (UniqueName: \"kubernetes.io/projected/5b13a079-9236-4642-b193-77b4903352c0-kube-api-access-rqtcf\") pod \"maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:44.757759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.757693    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b13a079-9236-4642-b193-77b4903352c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:44.859474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.859442    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b13a079-9236-4642-b193-77b4903352c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:44.859695 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.859550    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqtcf\" (UniqueName: \"kubernetes.io/projected/5b13a079-9236-4642-b193-77b4903352c0-kube-api-access-rqtcf\") pod \"maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:44.859793 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.859776    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b13a079-9236-4642-b193-77b4903352c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:44.860462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.860435    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b13a079-9236-4642-b193-77b4903352c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:44.862199 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.862178    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b13a079-9236-4642-b193-77b4903352c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:44.882859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.882796    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqtcf\" (UniqueName: \"kubernetes.io/projected/5b13a079-9236-4642-b193-77b4903352c0-kube-api-access-rqtcf\") pod \"maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:44.943568 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:44.943535    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:45.203171 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:45.202993    4555 scope.go:122] "RemoveContainer" containerID="302a52c0f3004db2f796ee96b4f15e257cd845ec298a0bf426dcad19ef59dd5b"
Oct 30 23:55:45.245687 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:45.245653    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"]
Oct 30 23:55:45.562760 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:45.562709    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98" event={"ID":"5b13a079-9236-4642-b193-77b4903352c0","Type":"ContainerStarted","Data":"08d46c81c4c7df70bdb4865fc26a314d75d69e8cbcd0beee38a17eb5ff53f70b"}
Oct 30 23:55:46.645359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:46.645322    4555 generic.go:356] "Generic (PLEG): container finished" podID="5b13a079-9236-4642-b193-77b4903352c0" containerID="a08bf4d2726289398ed7811891da48bc04db7c7f59954a41ec808539d9607f9b" exitCode=0
Oct 30 23:55:46.645813 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:46.645548    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98" event={"ID":"5b13a079-9236-4642-b193-77b4903352c0","Type":"ContainerDied","Data":"a08bf4d2726289398ed7811891da48bc04db7c7f59954a41ec808539d9607f9b"}
Oct 30 23:55:46.649038 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:46.649002    4555 generic.go:356] "Generic (PLEG): container finished" podID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerID="f53ad7082626bd2c240a7c48e964b2e06bfdebec9e7bbcabeeb8f5e1e8cc67c1" exitCode=7
Oct 30 23:55:46.649274 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:46.649256    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9" event={"ID":"767dcd83-f5fd-46a7-ad3c-73361d87316e","Type":"ContainerDied","Data":"f53ad7082626bd2c240a7c48e964b2e06bfdebec9e7bbcabeeb8f5e1e8cc67c1"}
Oct 30 23:55:46.649350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:46.649321    4555 scope.go:122] "RemoveContainer" containerID="302a52c0f3004db2f796ee96b4f15e257cd845ec298a0bf426dcad19ef59dd5b"
Oct 30 23:55:46.649980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:46.649956    4555 scope.go:122] "RemoveContainer" containerID="f53ad7082626bd2c240a7c48e964b2e06bfdebec9e7bbcabeeb8f5e1e8cc67c1"
Oct 30 23:55:46.650519 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:55:46.650490    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-sqrn9_odh-ai-gateway-infra(767dcd83-f5fd-46a7-ad3c-73361d87316e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e"
Oct 30 23:55:47.732598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:47.732567    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"]
Oct 30 23:55:47.899166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:47.899130    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:47.979890 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:47.979865    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:48.041327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.041291    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b13a079-9236-4642-b193-77b4903352c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b13a079-9236-4642-b193-77b4903352c0-maas-api-xks-ca\") pod \"5b13a079-9236-4642-b193-77b4903352c0\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") "
Oct 30 23:55:48.041531 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.041466    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b13a079-9236-4642-b193-77b4903352c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b13a079-9236-4642-b193-77b4903352c0-maas-api-openshift-ca\") pod \"5b13a079-9236-4642-b193-77b4903352c0\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") "
Oct 30 23:55:48.041684 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.041669    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b13a079-9236-4642-b193-77b4903352c0-kube-api-access-rqtcf\" (UniqueName: \"kubernetes.io/projected/5b13a079-9236-4642-b193-77b4903352c0-kube-api-access-rqtcf\") pod \"5b13a079-9236-4642-b193-77b4903352c0\" (UID: \"5b13a079-9236-4642-b193-77b4903352c0\") "
Oct 30 23:55:48.042081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.042050    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b13a079-9236-4642-b193-77b4903352c0-maas-api-openshift-ca" pod "5b13a079-9236-4642-b193-77b4903352c0" (UID: "5b13a079-9236-4642-b193-77b4903352c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:55:48.042740 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.042560    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b13a079-9236-4642-b193-77b4903352c0-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:48.048626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.045573    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b13a079-9236-4642-b193-77b4903352c0-kube-api-access-rqtcf" pod "5b13a079-9236-4642-b193-77b4903352c0" (UID: "5b13a079-9236-4642-b193-77b4903352c0"). InnerVolumeSpecName "kube-api-access-rqtcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:55:48.048626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.045660    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b13a079-9236-4642-b193-77b4903352c0-maas-api-xks-ca" pod "5b13a079-9236-4642-b193-77b4903352c0" (UID: "5b13a079-9236-4642-b193-77b4903352c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:55:48.143258 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.143225    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-xks-ca\") pod \"767dcd83-f5fd-46a7-ad3c-73361d87316e\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") "
Oct 30 23:55:48.143543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.143523    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767dcd83-f5fd-46a7-ad3c-73361d87316e-kube-api-access-8lqn9\" (UniqueName: \"kubernetes.io/projected/767dcd83-f5fd-46a7-ad3c-73361d87316e-kube-api-access-8lqn9\") pod \"767dcd83-f5fd-46a7-ad3c-73361d87316e\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") "
Oct 30 23:55:48.143652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.143629    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-openshift-ca\") pod \"767dcd83-f5fd-46a7-ad3c-73361d87316e\" (UID: \"767dcd83-f5fd-46a7-ad3c-73361d87316e\") "
Oct 30 23:55:48.144007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.143976    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-openshift-ca" pod "767dcd83-f5fd-46a7-ad3c-73361d87316e" (UID: "767dcd83-f5fd-46a7-ad3c-73361d87316e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:55:48.144485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.144464    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:48.144568 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.144490    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b13a079-9236-4642-b193-77b4903352c0-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:48.144568 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.144506    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqtcf\" (UniqueName: \"kubernetes.io/projected/5b13a079-9236-4642-b193-77b4903352c0-kube-api-access-rqtcf\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:48.145570 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.145549    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-xks-ca" pod "767dcd83-f5fd-46a7-ad3c-73361d87316e" (UID: "767dcd83-f5fd-46a7-ad3c-73361d87316e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:55:48.145841 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.145816    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767dcd83-f5fd-46a7-ad3c-73361d87316e-kube-api-access-8lqn9" pod "767dcd83-f5fd-46a7-ad3c-73361d87316e" (UID: "767dcd83-f5fd-46a7-ad3c-73361d87316e"). InnerVolumeSpecName "kube-api-access-8lqn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:55:48.246254 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.246167    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767dcd83-f5fd-46a7-ad3c-73361d87316e-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:48.246254 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.246202    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lqn9\" (UniqueName: \"kubernetes.io/projected/767dcd83-f5fd-46a7-ad3c-73361d87316e-kube-api-access-8lqn9\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:55:48.672053 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.672013    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98" event={"ID":"5b13a079-9236-4642-b193-77b4903352c0","Type":"ContainerDied","Data":"08d46c81c4c7df70bdb4865fc26a314d75d69e8cbcd0beee38a17eb5ff53f70b"}
Oct 30 23:55:48.672262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.672083    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08d46c81c4c7df70bdb4865fc26a314d75d69e8cbcd0beee38a17eb5ff53f70b"
Oct 30 23:55:48.672262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.672168    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"
Oct 30 23:55:48.676244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.676221    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"
Oct 30 23:55:48.676359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.676246    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9" event={"ID":"767dcd83-f5fd-46a7-ad3c-73361d87316e","Type":"ContainerDied","Data":"7ea2f045088640be943174eeb99833294093e4f2191ec676aecb3ab4958f4c23"}
Oct 30 23:55:48.676359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.676316    4555 scope.go:122] "RemoveContainer" containerID="f53ad7082626bd2c240a7c48e964b2e06bfdebec9e7bbcabeeb8f5e1e8cc67c1"
Oct 30 23:55:48.715145 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.715121    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"]
Oct 30 23:55:48.720740 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:48.720714    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-sqrn9"]
Oct 30 23:55:49.213543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:49.213498    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e" path="/var/lib/kubelet/pods/767dcd83-f5fd-46a7-ad3c-73361d87316e/volumes"
Oct 30 23:55:50.578396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.573440    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"]
Oct 30 23:55:50.650442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650363    4555 cpu_manager.go:414] "containerMap: removing container" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerName="revoke-keys"
Oct 30 23:55:50.650442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650425    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerName="revoke-keys"
Oct 30 23:55:50.650863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650465    4555 cpu_manager.go:414] "containerMap: removing container" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerName="revoke-keys"
Oct 30 23:55:50.650863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650473    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerName="revoke-keys"
Oct 30 23:55:50.650863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650507    4555 cpu_manager.go:414] "containerMap: removing container" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerName="revoke-keys"
Oct 30 23:55:50.650863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650514    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerName="revoke-keys"
Oct 30 23:55:50.650863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650552    4555 cpu_manager.go:414] "containerMap: removing container" podUID="5b13a079-9236-4642-b193-77b4903352c0" containerName="revoke-keys"
Oct 30 23:55:50.650863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650559    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b13a079-9236-4642-b193-77b4903352c0" containerName="revoke-keys"
Oct 30 23:55:50.650863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650827    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerName="revoke-keys"
Oct 30 23:55:50.650863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650846    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerName="revoke-keys"
Oct 30 23:55:50.650863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.650864    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b13a079-9236-4642-b193-77b4903352c0" containerName="revoke-keys"
Oct 30 23:55:50.669436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.667615    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"]
Oct 30 23:55:50.669436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.667733    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:50.675665 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.673903    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:55:50.675665 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.674041    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:55:50.744553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.744514    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-68tn8\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:50.744764 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.744742    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7zp9\" (UniqueName: \"kubernetes.io/projected/5668fe30-897d-4435-8d2a-4508a603580e-kube-api-access-f7zp9\") pod \"maas-api-revoke-sub-91354e50-68tn8\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:50.744882 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.744866    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-68tn8\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:50.846923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.846838    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-68tn8\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:50.847073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.847022    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zp9\" (UniqueName: \"kubernetes.io/projected/5668fe30-897d-4435-8d2a-4508a603580e-kube-api-access-f7zp9\") pod \"maas-api-revoke-sub-91354e50-68tn8\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:50.847156 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.847141    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-68tn8\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:50.847869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.847845    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-68tn8\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:50.849469 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.849442    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-68tn8\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:50.859268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:50.859234    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zp9\" (UniqueName: \"kubernetes.io/projected/5668fe30-897d-4435-8d2a-4508a603580e-kube-api-access-f7zp9\") pod \"maas-api-revoke-sub-91354e50-68tn8\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:51.002983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:51.002948    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:55:51.219410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:51.219305    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:55:51.369777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:51.369745    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"]
Oct 30 23:55:51.720153 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:51.720115    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" event={"ID":"5668fe30-897d-4435-8d2a-4508a603580e","Type":"ContainerStarted","Data":"611fbf2daa1f648cb233dc95f5ed11720876525245724c1ed3a2454424c2ad98"}
Oct 30 23:55:52.568801 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:52.564477    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"]
Oct 30 23:55:52.580409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:52.577409    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2b4009-grqmx"]
Oct 30 23:55:52.822667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:52.821920    4555 generic.go:356] "Generic (PLEG): container finished" podID="5668fe30-897d-4435-8d2a-4508a603580e" containerID="c6c03dd967b578c8d28f621a9fc890c0b30b6062ce07290f4263605457cd6eb8" exitCode=7
Oct 30 23:55:52.822667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:52.822031    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" event={"ID":"5668fe30-897d-4435-8d2a-4508a603580e","Type":"ContainerDied","Data":"c6c03dd967b578c8d28f621a9fc890c0b30b6062ce07290f4263605457cd6eb8"}
Oct 30 23:55:52.822667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:52.822596    4555 scope.go:122] "RemoveContainer" containerID="c6c03dd967b578c8d28f621a9fc890c0b30b6062ce07290f4263605457cd6eb8"
Oct 30 23:55:53.212351 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:53.212319    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11457ba3-82a3-4994-a1ab-29ffb4046b04" path="/var/lib/kubelet/pods/11457ba3-82a3-4994-a1ab-29ffb4046b04/volumes"
Oct 30 23:55:53.834597 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:53.834560    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" event={"ID":"5668fe30-897d-4435-8d2a-4508a603580e","Type":"ContainerStarted","Data":"456df3478d38460bbe1f0a1983a94a9fd52753bb973a967011d72b79d702bd1d"}
Oct 30 23:55:53.862036 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:53.861975    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" podStartSLOduration=3.861956896 podStartE2EDuration="3.861956896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:53.857829165 +0000 UTC m=+2212.784466741" watchObservedRunningTime="2026-10-30 23:55:53.861956896 +0000 UTC m=+2212.788594479"
Oct 30 23:55:54.909405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:54.909333    4555 generic.go:356] "Generic (PLEG): container finished" podID="5668fe30-897d-4435-8d2a-4508a603580e" containerID="456df3478d38460bbe1f0a1983a94a9fd52753bb973a967011d72b79d702bd1d" exitCode=7
Oct 30 23:55:54.910251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:54.910224    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" event={"ID":"5668fe30-897d-4435-8d2a-4508a603580e","Type":"ContainerDied","Data":"456df3478d38460bbe1f0a1983a94a9fd52753bb973a967011d72b79d702bd1d"}
Oct 30 23:55:54.910508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:54.910492    4555 scope.go:122] "RemoveContainer" containerID="c6c03dd967b578c8d28f621a9fc890c0b30b6062ce07290f4263605457cd6eb8"
Oct 30 23:55:54.911123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:54.911105    4555 scope.go:122] "RemoveContainer" containerID="456df3478d38460bbe1f0a1983a94a9fd52753bb973a967011d72b79d702bd1d"
Oct 30 23:55:54.911758 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:55:54.911733    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-68tn8_odh-ai-gateway-infra(5668fe30-897d-4435-8d2a-4508a603580e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" podUID="5668fe30-897d-4435-8d2a-4508a603580e"
Oct 30 23:55:55.920654 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:55.920622    4555 scope.go:122] "RemoveContainer" containerID="456df3478d38460bbe1f0a1983a94a9fd52753bb973a967011d72b79d702bd1d"
Oct 30 23:55:55.921062 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:55:55.921036    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-68tn8_odh-ai-gateway-infra(5668fe30-897d-4435-8d2a-4508a603580e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" podUID="5668fe30-897d-4435-8d2a-4508a603580e"
Oct 30 23:55:57.449044 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:55:57.449006    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:55:58.461102 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.461069    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"]
Oct 30 23:55:58.462034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.462012    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="767dcd83-f5fd-46a7-ad3c-73361d87316e" containerName="revoke-keys"
Oct 30 23:55:58.467715 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.467695    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 30 23:55:58.483308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.483262    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"]
Oct 30 23:55:58.487959 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.486757    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6b38a4\""
Oct 30 23:55:58.615823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.615773    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8c7cd4-5bea-4777-ba9d-868d9096648d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp\" (UID: \"2c8c7cd4-5bea-4777-ba9d-868d9096648d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 30 23:55:58.616171 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.615829    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmbqk\" (UniqueName: \"kubernetes.io/projected/2c8c7cd4-5bea-4777-ba9d-868d9096648d-kube-api-access-qmbqk\") pod \"payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp\" (UID: \"2c8c7cd4-5bea-4777-ba9d-868d9096648d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 30 23:55:58.728902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.728640    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8c7cd4-5bea-4777-ba9d-868d9096648d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp\" (UID: \"2c8c7cd4-5bea-4777-ba9d-868d9096648d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 30 23:55:58.733928 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.733891    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmbqk\" (UniqueName: \"kubernetes.io/projected/2c8c7cd4-5bea-4777-ba9d-868d9096648d-kube-api-access-qmbqk\") pod \"payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp\" (UID: \"2c8c7cd4-5bea-4777-ba9d-868d9096648d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 30 23:55:58.734773 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.734728    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8c7cd4-5bea-4777-ba9d-868d9096648d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp\" (UID: \"2c8c7cd4-5bea-4777-ba9d-868d9096648d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 30 23:55:58.816505 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:58.816471    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmbqk\" (UniqueName: \"kubernetes.io/projected/2c8c7cd4-5bea-4777-ba9d-868d9096648d-kube-api-access-qmbqk\") pod \"payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp\" (UID: \"2c8c7cd4-5bea-4777-ba9d-868d9096648d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 30 23:55:59.097398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:59.095300    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 30 23:55:59.414328 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:55:59.414282    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8c7cd4_5bea_4777_ba9d_868d9096648d.slice/crio-c8bdf7f7dc7cb6dd7793e3c472b54cd69f22da449ea2188a0a86141149fa6cd8 WatchSource:0}: Error finding container c8bdf7f7dc7cb6dd7793e3c472b54cd69f22da449ea2188a0a86141149fa6cd8: Status 404 returned error can't find the container with id c8bdf7f7dc7cb6dd7793e3c472b54cd69f22da449ea2188a0a86141149fa6cd8
Oct 30 23:55:59.416777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:59.416753    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"]
Oct 30 23:55:59.989708 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:59.989656    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp" event={"ID":"2c8c7cd4-5bea-4777-ba9d-868d9096648d","Type":"ContainerStarted","Data":"7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237"}
Oct 30 23:55:59.990154 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:59.989730    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp" event={"ID":"2c8c7cd4-5bea-4777-ba9d-868d9096648d","Type":"ContainerStarted","Data":"c8bdf7f7dc7cb6dd7793e3c472b54cd69f22da449ea2188a0a86141149fa6cd8"}
Oct 30 23:55:59.990154 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:55:59.989759    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 30 23:56:00.029804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:00.029718    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp" podStartSLOduration=2.029697458 podStartE2EDuration="2.029697458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:00.02698787 +0000 UTC m=+2218.953625453" watchObservedRunningTime="2026-10-30 23:56:00.029697458 +0000 UTC m=+2218.956335040"
Oct 30 23:56:03.223901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:03.223857    4555 scope.go:122] "RemoveContainer" containerID="9cc1708765dea9f34ec30d04a6d53ec041e9c8f77a0d42af5a1397d42fa49985"
Oct 30 23:56:06.860747 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:06.860719    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"]
Oct 30 23:56:06.873405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:06.873356    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:06.885777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:06.885360    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"]
Oct 30 23:56:06.954650 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:06.954616    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82d449-2hk52\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:06.954858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:06.954833    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82d449-2hk52\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:06.954969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:06.954951    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xwvb\" (UniqueName: \"kubernetes.io/projected/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-kube-api-access-8xwvb\") pod \"maas-api-revoke-keys-e2e-inf-a-82d449-2hk52\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:07.056036 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:07.056003    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82d449-2hk52\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:07.056226 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:07.056148    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82d449-2hk52\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:07.056226 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:07.056212    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwvb\" (UniqueName: \"kubernetes.io/projected/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-kube-api-access-8xwvb\") pod \"maas-api-revoke-keys-e2e-inf-a-82d449-2hk52\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:07.056825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:07.056798    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82d449-2hk52\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:07.058929 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:07.058889    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-82d449-2hk52\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:07.070559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:07.070534    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwvb\" (UniqueName: \"kubernetes.io/projected/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-kube-api-access-8xwvb\") pod \"maas-api-revoke-keys-e2e-inf-a-82d449-2hk52\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:07.198124 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:07.198043    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:07.501269 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:56:07.500730    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3109ee6_6479_4a2a_bc3c_d7b77734d8a5.slice/crio-8c1c88ae1a68eb2449e275eca921c426db86449b6c4ee50c48a3408912988571 WatchSource:0}: Error finding container 8c1c88ae1a68eb2449e275eca921c426db86449b6c4ee50c48a3408912988571: Status 404 returned error can't find the container with id 8c1c88ae1a68eb2449e275eca921c426db86449b6c4ee50c48a3408912988571
Oct 30 23:56:07.503195 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:07.503050    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"]
Oct 30 23:56:08.075132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:08.075047    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52" event={"ID":"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5","Type":"ContainerStarted","Data":"3f1b5818eba2bf2586500728d61c541b0dfcaebae1837daaa610d8d34ba3f457"}
Oct 30 23:56:08.075132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:08.075120    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52" event={"ID":"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5","Type":"ContainerStarted","Data":"8c1c88ae1a68eb2449e275eca921c426db86449b6c4ee50c48a3408912988571"}
Oct 30 23:56:08.109299 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:08.107516    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52" podStartSLOduration=2.107497654 podStartE2EDuration="2.107497654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:08.10620606 +0000 UTC m=+2227.032843633" watchObservedRunningTime="2026-10-30 23:56:08.107497654 +0000 UTC m=+2227.034135232"
Oct 30 23:56:08.201273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:08.201240    4555 scope.go:122] "RemoveContainer" containerID="456df3478d38460bbe1f0a1983a94a9fd52753bb973a967011d72b79d702bd1d"
Oct 30 23:56:09.087485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:09.087450    4555 generic.go:356] "Generic (PLEG): container finished" podID="f3109ee6-6479-4a2a-bc3c-d7b77734d8a5" containerID="3f1b5818eba2bf2586500728d61c541b0dfcaebae1837daaa610d8d34ba3f457" exitCode=0
Oct 30 23:56:09.088011 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:09.087717    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52" event={"ID":"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5","Type":"ContainerDied","Data":"3f1b5818eba2bf2586500728d61c541b0dfcaebae1837daaa610d8d34ba3f457"}
Oct 30 23:56:09.090418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:09.090347    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" event={"ID":"5668fe30-897d-4435-8d2a-4508a603580e","Type":"ContainerStarted","Data":"248004c8d5653ffccc6afebe94ac181250bad5f60e09a53e441490bfbf1cac5f"}
Oct 30 23:56:10.105856 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.105822    4555 generic.go:356] "Generic (PLEG): container finished" podID="5668fe30-897d-4435-8d2a-4508a603580e" containerID="248004c8d5653ffccc6afebe94ac181250bad5f60e09a53e441490bfbf1cac5f" exitCode=7
Oct 30 23:56:10.106363 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.106229    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" event={"ID":"5668fe30-897d-4435-8d2a-4508a603580e","Type":"ContainerDied","Data":"248004c8d5653ffccc6afebe94ac181250bad5f60e09a53e441490bfbf1cac5f"}
Oct 30 23:56:10.106363 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.106312    4555 scope.go:122] "RemoveContainer" containerID="456df3478d38460bbe1f0a1983a94a9fd52753bb973a967011d72b79d702bd1d"
Oct 30 23:56:10.106918 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.106901    4555 scope.go:122] "RemoveContainer" containerID="248004c8d5653ffccc6afebe94ac181250bad5f60e09a53e441490bfbf1cac5f"
Oct 30 23:56:10.107358 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:56:10.107334    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-68tn8_odh-ai-gateway-infra(5668fe30-897d-4435-8d2a-4508a603580e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" podUID="5668fe30-897d-4435-8d2a-4508a603580e"
Oct 30 23:56:10.154683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.154644    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"]
Oct 30 23:56:10.316804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.316779    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:10.497795 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.497704    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-kube-api-access-8xwvb\" (UniqueName: \"kubernetes.io/projected/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-kube-api-access-8xwvb\") pod \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") "
Oct 30 23:56:10.498081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.498056    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-openshift-ca\") pod \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") "
Oct 30 23:56:10.498227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.498210    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-xks-ca\") pod \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\" (UID: \"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5\") "
Oct 30 23:56:10.498548 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.498520    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-openshift-ca" pod "f3109ee6-6479-4a2a-bc3c-d7b77734d8a5" (UID: "f3109ee6-6479-4a2a-bc3c-d7b77734d8a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:10.500341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.500318    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:10.502558 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.501835    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-xks-ca" pod "f3109ee6-6479-4a2a-bc3c-d7b77734d8a5" (UID: "f3109ee6-6479-4a2a-bc3c-d7b77734d8a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:10.518186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.518143    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-kube-api-access-8xwvb" pod "f3109ee6-6479-4a2a-bc3c-d7b77734d8a5" (UID: "f3109ee6-6479-4a2a-bc3c-d7b77734d8a5"). InnerVolumeSpecName "kube-api-access-8xwvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:10.601753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.601722    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xwvb\" (UniqueName: \"kubernetes.io/projected/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-kube-api-access-8xwvb\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:10.601753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:10.601748    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:11.002035 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:11.002001    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 30 23:56:11.474624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:11.467542    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"
Oct 30 23:56:11.725442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:11.724683    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"]
Oct 30 23:56:11.725442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:11.724730    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52" event={"ID":"f3109ee6-6479-4a2a-bc3c-d7b77734d8a5","Type":"ContainerDied","Data":"8c1c88ae1a68eb2449e275eca921c426db86449b6c4ee50c48a3408912988571"}
Oct 30 23:56:11.725442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:11.724754    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c1c88ae1a68eb2449e275eca921c426db86449b6c4ee50c48a3408912988571"
Oct 30 23:56:11.725442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:11.724774    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f026c1-v4t98"]
Oct 30 23:56:12.042193 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.042171    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:56:12.214549 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.214511    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5668fe30-897d-4435-8d2a-4508a603580e-kube-api-access-f7zp9\" (UniqueName: \"kubernetes.io/projected/5668fe30-897d-4435-8d2a-4508a603580e-kube-api-access-f7zp9\") pod \"5668fe30-897d-4435-8d2a-4508a603580e\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") "
Oct 30 23:56:12.214981 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.214956    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-openshift-ca\") pod \"5668fe30-897d-4435-8d2a-4508a603580e\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") "
Oct 30 23:56:12.215080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.214998    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-xks-ca\") pod \"5668fe30-897d-4435-8d2a-4508a603580e\" (UID: \"5668fe30-897d-4435-8d2a-4508a603580e\") "
Oct 30 23:56:12.215420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.215357    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-openshift-ca" pod "5668fe30-897d-4435-8d2a-4508a603580e" (UID: "5668fe30-897d-4435-8d2a-4508a603580e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:12.215701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.215685    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:12.217422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.217399    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-xks-ca" pod "5668fe30-897d-4435-8d2a-4508a603580e" (UID: "5668fe30-897d-4435-8d2a-4508a603580e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:12.217488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.217424    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5668fe30-897d-4435-8d2a-4508a603580e-kube-api-access-f7zp9" pod "5668fe30-897d-4435-8d2a-4508a603580e" (UID: "5668fe30-897d-4435-8d2a-4508a603580e"). InnerVolumeSpecName "kube-api-access-f7zp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:12.317126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.317095    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5668fe30-897d-4435-8d2a-4508a603580e-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:12.317291 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.317129    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7zp9\" (UniqueName: \"kubernetes.io/projected/5668fe30-897d-4435-8d2a-4508a603580e-kube-api-access-f7zp9\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:12.542714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.542681    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"
Oct 30 23:56:12.543216 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.542712    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8" event={"ID":"5668fe30-897d-4435-8d2a-4508a603580e","Type":"ContainerDied","Data":"611fbf2daa1f648cb233dc95f5ed11720876525245724c1ed3a2454424c2ad98"}
Oct 30 23:56:12.543216 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.542776    4555 scope.go:122] "RemoveContainer" containerID="248004c8d5653ffccc6afebe94ac181250bad5f60e09a53e441490bfbf1cac5f"
Oct 30 23:56:12.598645 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.592309    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"]
Oct 30 23:56:12.612780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:12.612747    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-68tn8"]
Oct 30 23:56:13.211479 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:13.211435    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5668fe30-897d-4435-8d2a-4508a603580e" path="/var/lib/kubelet/pods/5668fe30-897d-4435-8d2a-4508a603580e/volumes"
Oct 30 23:56:13.212120 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:13.212095    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b13a079-9236-4642-b193-77b4903352c0" path="/var/lib/kubelet/pods/5b13a079-9236-4642-b193-77b4903352c0/volumes"
Oct 30 23:56:17.195606 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:17.195573    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"]
Oct 30 23:56:20.722628 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.722584    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"]
Oct 30 23:56:20.723510 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.723483    4555 cpu_manager.go:414] "containerMap: removing container" podUID="5668fe30-897d-4435-8d2a-4508a603580e" containerName="revoke-keys"
Oct 30 23:56:20.723554 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.723511    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5668fe30-897d-4435-8d2a-4508a603580e" containerName="revoke-keys"
Oct 30 23:56:20.723554 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.723522    4555 cpu_manager.go:414] "containerMap: removing container" podUID="5668fe30-897d-4435-8d2a-4508a603580e" containerName="revoke-keys"
Oct 30 23:56:20.723554 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.723527    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5668fe30-897d-4435-8d2a-4508a603580e" containerName="revoke-keys"
Oct 30 23:56:20.723664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.723555    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f3109ee6-6479-4a2a-bc3c-d7b77734d8a5" containerName="revoke-keys"
Oct 30 23:56:20.723664 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.723561    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3109ee6-6479-4a2a-bc3c-d7b77734d8a5" containerName="revoke-keys"
Oct 30 23:56:20.723734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.723687    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="5668fe30-897d-4435-8d2a-4508a603580e" containerName="revoke-keys"
Oct 30 23:56:20.723734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.723695    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="5668fe30-897d-4435-8d2a-4508a603580e" containerName="revoke-keys"
Oct 30 23:56:20.723734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.723706    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3109ee6-6479-4a2a-bc3c-d7b77734d8a5" containerName="revoke-keys"
Oct 30 23:56:20.723734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.723714    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="5668fe30-897d-4435-8d2a-4508a603580e" containerName="revoke-keys"
Oct 30 23:56:20.729167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.729148    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:20.743883 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.736181    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:56:20.743883 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.736192    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:56:20.744425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.744143    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"]
Oct 30 23:56:20.872709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.870956    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcfzc\" (UniqueName: \"kubernetes.io/projected/48a90375-8b99-4bc0-9afe-075ed088493a-kube-api-access-qcfzc\") pod \"maas-api-revoke-sub-91354e50-4hnnj\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:20.872709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.871144    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-4hnnj\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:20.872709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.871281    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-4hnnj\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:20.973148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.973062    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcfzc\" (UniqueName: \"kubernetes.io/projected/48a90375-8b99-4bc0-9afe-075ed088493a-kube-api-access-qcfzc\") pod \"maas-api-revoke-sub-91354e50-4hnnj\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:20.973326 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.973171    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-4hnnj\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:20.973326 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.973255    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-4hnnj\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:20.973945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.973918    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-4hnnj\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:20.976006 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.975984    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-4hnnj\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:20.989302 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:20.989271    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcfzc\" (UniqueName: \"kubernetes.io/projected/48a90375-8b99-4bc0-9afe-075ed088493a-kube-api-access-qcfzc\") pod \"maas-api-revoke-sub-91354e50-4hnnj\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:21.060268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:21.060239    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:21.364422 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:56:21.364347    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a90375_8b99_4bc0_9afe_075ed088493a.slice/crio-e1a3a9080ea4d929a07435baa837dc69a5a6fc7d2157c60fa876d884771a898f WatchSource:0}: Error finding container e1a3a9080ea4d929a07435baa837dc69a5a6fc7d2157c60fa876d884771a898f: Status 404 returned error can't find the container with id e1a3a9080ea4d929a07435baa837dc69a5a6fc7d2157c60fa876d884771a898f
Oct 30 23:56:21.365657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:21.365626    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"]
Oct 30 23:56:21.687298 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:21.687214    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj" event={"ID":"48a90375-8b99-4bc0-9afe-075ed088493a","Type":"ContainerStarted","Data":"e1a3a9080ea4d929a07435baa837dc69a5a6fc7d2157c60fa876d884771a898f"}
Oct 30 23:56:22.698903 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:22.698818    4555 generic.go:356] "Generic (PLEG): container finished" podID="48a90375-8b99-4bc0-9afe-075ed088493a" containerID="c0e9162b790c295b35ed00618497b2f8bd8f41012464dea08556016f75040d77" exitCode=7
Oct 30 23:56:22.699290 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:22.698907    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj" event={"ID":"48a90375-8b99-4bc0-9afe-075ed088493a","Type":"ContainerDied","Data":"c0e9162b790c295b35ed00618497b2f8bd8f41012464dea08556016f75040d77"}
Oct 30 23:56:22.699290 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:22.699273    4555 scope.go:122] "RemoveContainer" containerID="c0e9162b790c295b35ed00618497b2f8bd8f41012464dea08556016f75040d77"
Oct 30 23:56:23.713211 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:23.713180    4555 generic.go:356] "Generic (PLEG): container finished" podID="48a90375-8b99-4bc0-9afe-075ed088493a" containerID="523d6b1297853c12374e10b52ef5f4e63d8de65fda3f4eacd2ed410731fd3ef2" exitCode=7
Oct 30 23:56:23.713674 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:23.713266    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj" event={"ID":"48a90375-8b99-4bc0-9afe-075ed088493a","Type":"ContainerDied","Data":"523d6b1297853c12374e10b52ef5f4e63d8de65fda3f4eacd2ed410731fd3ef2"}
Oct 30 23:56:23.713674 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:23.713340    4555 scope.go:122] "RemoveContainer" containerID="c0e9162b790c295b35ed00618497b2f8bd8f41012464dea08556016f75040d77"
Oct 30 23:56:23.713674 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:23.713600    4555 scope.go:122] "RemoveContainer" containerID="523d6b1297853c12374e10b52ef5f4e63d8de65fda3f4eacd2ed410731fd3ef2"
Oct 30 23:56:23.713946 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:56:23.713922    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-4hnnj_odh-ai-gateway-infra(48a90375-8b99-4bc0-9afe-075ed088493a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj" podUID="48a90375-8b99-4bc0-9afe-075ed088493a"
Oct 30 23:56:24.351974 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:24.351935    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"]
Oct 30 23:56:24.355978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:24.355903    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk" podUID="0ebbcb45-d81e-43a4-b925-bd65f76e3c9b" containerName="maas-api" containerID="cri-o://bb760cf055b0d990106410be8175ce04043cbe2b3734459d2dd1b4f65eff09cd" gracePeriod=30
Oct 30 23:56:25.132486 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.132441    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"]
Oct 30 23:56:25.178410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.176271    4555 scope.go:122] "RemoveContainer" containerID="523d6b1297853c12374e10b52ef5f4e63d8de65fda3f4eacd2ed410731fd3ef2"
Oct 30 23:56:25.181687 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.181253    4555 generic.go:356] "Generic (PLEG): container finished" podID="0ebbcb45-d81e-43a4-b925-bd65f76e3c9b" containerID="bb760cf055b0d990106410be8175ce04043cbe2b3734459d2dd1b4f65eff09cd" exitCode=0
Oct 30 23:56:25.181687 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.181423    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk" event={"ID":"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b","Type":"ContainerDied","Data":"bb760cf055b0d990106410be8175ce04043cbe2b3734459d2dd1b4f65eff09cd"}
Oct 30 23:56:25.243423 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:56:25.232906    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-4hnnj_odh-ai-gateway-infra(48a90375-8b99-4bc0-9afe-075ed088493a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj" podUID="48a90375-8b99-4bc0-9afe-075ed088493a"
Oct 30 23:56:25.267396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.258010    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:56:25.352895 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.352862    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-82d449-2hk52"]
Oct 30 23:56:25.379212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.379114    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-metrics-certs\") pod \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") "
Oct 30 23:56:25.379471 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.379280    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-maas-api-tls\") pod \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") "
Oct 30 23:56:25.379471 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.379315    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-kube-api-access-zw7w8\" (UniqueName: \"kubernetes.io/projected/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-kube-api-access-zw7w8\") pod \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") "
Oct 30 23:56:25.379471 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.379346    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-openshift-service-ca\") pod \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\" (UID: \"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b\") "
Oct 30 23:56:25.380500 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.380180    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-openshift-service-ca" pod "0ebbcb45-d81e-43a4-b925-bd65f76e3c9b" (UID: "0ebbcb45-d81e-43a4-b925-bd65f76e3c9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:25.380613 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.380499    4555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-openshift-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:25.382280 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.382255    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-metrics-certs" pod "0ebbcb45-d81e-43a4-b925-bd65f76e3c9b" (UID: "0ebbcb45-d81e-43a4-b925-bd65f76e3c9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:25.382667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.382639    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-maas-api-tls" pod "0ebbcb45-d81e-43a4-b925-bd65f76e3c9b" (UID: "0ebbcb45-d81e-43a4-b925-bd65f76e3c9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:25.384204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.383998    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"]
Oct 30 23:56:25.384204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.384158    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-kube-api-access-zw7w8" pod "0ebbcb45-d81e-43a4-b925-bd65f76e3c9b" (UID: "0ebbcb45-d81e-43a4-b925-bd65f76e3c9b"). InnerVolumeSpecName "kube-api-access-zw7w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:25.385343 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.385320    4555 cpu_manager.go:414] "containerMap: removing container" podUID="5668fe30-897d-4435-8d2a-4508a603580e" containerName="revoke-keys"
Oct 30 23:56:25.385343 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.385342    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5668fe30-897d-4435-8d2a-4508a603580e" containerName="revoke-keys"
Oct 30 23:56:25.385528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.385446    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0ebbcb45-d81e-43a4-b925-bd65f76e3c9b" containerName="maas-api"
Oct 30 23:56:25.385528 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.385455    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebbcb45-d81e-43a4-b925-bd65f76e3c9b" containerName="maas-api"
Oct 30 23:56:25.385676 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.385660    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebbcb45-d81e-43a4-b925-bd65f76e3c9b" containerName="maas-api"
Oct 30 23:56:25.391580 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.390950    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.392170 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.392150    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"]
Oct 30 23:56:25.406178 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.406154    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6b38a4-serving-cert\""
Oct 30 23:56:25.482448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.482416    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-metrics-certs\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.482640 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.482622    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.482725 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.482670    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.482897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.482867    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfcww\" (UniqueName: \"kubernetes.io/projected/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-kube-api-access-cfcww\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.483014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.482992    4555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-metrics-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:25.483014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.483008    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-maas-api-tls\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:25.483095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.483021    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw7w8\" (UniqueName: \"kubernetes.io/projected/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b-kube-api-access-zw7w8\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:25.584356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.584313    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.584571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.584405    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.584638 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.584565    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfcww\" (UniqueName: \"kubernetes.io/projected/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-kube-api-access-cfcww\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.584724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.584694    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-metrics-certs\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.585287 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.585254    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.587814 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.587792    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-metrics-certs\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.587950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.587869    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.598149 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.598121    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfcww\" (UniqueName: \"kubernetes.io/projected/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-kube-api-access-cfcww\") pod \"maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:25.746645 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:25.746558    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:26.048390 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:56:26.048343    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5a1218_e055_4dca_9ee2_9285fcfd1fd5.slice/crio-dfc52574c3f994b8d22b837632b78a7b33c72bd7a54a48f6451d921abf917d7f WatchSource:0}: Error finding container dfc52574c3f994b8d22b837632b78a7b33c72bd7a54a48f6451d921abf917d7f: Status 404 returned error can't find the container with id dfc52574c3f994b8d22b837632b78a7b33c72bd7a54a48f6451d921abf917d7f
Oct 30 23:56:26.050175 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:26.050154    4555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:56:26.051464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:26.051438    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"]
Oct 30 23:56:26.326520 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:26.318928    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628" event={"ID":"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5","Type":"ContainerStarted","Data":"dfc52574c3f994b8d22b837632b78a7b33c72bd7a54a48f6451d921abf917d7f"}
Oct 30 23:56:26.330392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:26.330167    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk" event={"ID":"0ebbcb45-d81e-43a4-b925-bd65f76e3c9b","Type":"ContainerDied","Data":"1605c0e3abed68384a47022167ee5a428a4cefe3d2cc045a926639a29a1eb22f"}
Oct 30 23:56:26.330392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:26.330242    4555 scope.go:122] "RemoveContainer" containerID="bb760cf055b0d990106410be8175ce04043cbe2b3734459d2dd1b4f65eff09cd"
Oct 30 23:56:26.330620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:26.330427    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"
Oct 30 23:56:26.492201 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:26.492145    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"]
Oct 30 23:56:26.502414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:26.500821    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82d449-5fc7ccf965-6sffk"]
Oct 30 23:56:27.211527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:27.211490    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebbcb45-d81e-43a4-b925-bd65f76e3c9b" path="/var/lib/kubelet/pods/0ebbcb45-d81e-43a4-b925-bd65f76e3c9b/volumes"
Oct 30 23:56:27.212211 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:27.212191    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3109ee6-6479-4a2a-bc3c-d7b77734d8a5" path="/var/lib/kubelet/pods/f3109ee6-6479-4a2a-bc3c-d7b77734d8a5/volumes"
Oct 30 23:56:27.341578 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:27.341537    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628" event={"ID":"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5","Type":"ContainerStarted","Data":"c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd"}
Oct 30 23:56:27.342038 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:27.341607    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:27.366846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:27.366554    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628" podStartSLOduration=7.902621118 podStartE2EDuration="8.366539846s" totalImagesPullingTime="463.918728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:27.365545703 +0000 UTC m=+2246.292183283" watchObservedRunningTime="2026-10-30 23:56:27.366539846 +0000 UTC m=+2246.293177428"
Oct 30 23:56:27.449858 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:56:27.449819    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:56:30.571615 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:30.571581    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"]
Oct 30 23:56:32.232723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.232681    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4" podUID="61c076ae-c401-4b0a-b3e6-d358bf9dd00c" containerName="main" containerID="cri-o://369249b4a410e7b8f4215bf84ab464861f0f3d7296061ca591f4bea5c8ee9665" gracePeriod=45
Oct 30 23:56:32.391637 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.391319    4555 generic.go:356] "Generic (PLEG): container finished" podID="61c076ae-c401-4b0a-b3e6-d358bf9dd00c" containerID="369249b4a410e7b8f4215bf84ab464861f0f3d7296061ca591f4bea5c8ee9665" exitCode=0
Oct 30 23:56:32.391637 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.391480    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4" event={"ID":"61c076ae-c401-4b0a-b3e6-d358bf9dd00c","Type":"ContainerDied","Data":"369249b4a410e7b8f4215bf84ab464861f0f3d7296061ca591f4bea5c8ee9665"}
Oct 30 23:56:32.645727 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.645699    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:56:32.767644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.767606    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-kube-api-access-8lqq6\" (UniqueName: \"kubernetes.io/projected/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-kube-api-access-8lqq6\") pod \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") "
Oct 30 23:56:32.767930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.767899    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tls-certs\" (UniqueName: \"kubernetes.io/secret/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tls-certs\") pod \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") "
Oct 30 23:56:32.768055 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.768023    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tmp-dir\") pod \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") "
Oct 30 23:56:32.768161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.768144    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-dshm\") pod \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") "
Oct 30 23:56:32.768243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.768224    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-home\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-home\") pod \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") "
Oct 30 23:56:32.768349 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.768330    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-model-cache\") pod \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\" (UID: \"61c076ae-c401-4b0a-b3e6-d358bf9dd00c\") "
Oct 30 23:56:32.769519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.769493    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-model-cache" pod "61c076ae-c401-4b0a-b3e6-d358bf9dd00c" (UID: "61c076ae-c401-4b0a-b3e6-d358bf9dd00c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:32.770090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.770064    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tmp-dir" pod "61c076ae-c401-4b0a-b3e6-d358bf9dd00c" (UID: "61c076ae-c401-4b0a-b3e6-d358bf9dd00c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:32.770387 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.770342    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-home" pod "61c076ae-c401-4b0a-b3e6-d358bf9dd00c" (UID: "61c076ae-c401-4b0a-b3e6-d358bf9dd00c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:32.772947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.772926    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tls-certs" pod "61c076ae-c401-4b0a-b3e6-d358bf9dd00c" (UID: "61c076ae-c401-4b0a-b3e6-d358bf9dd00c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:32.773139 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.773113    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-kube-api-access-8lqq6" pod "61c076ae-c401-4b0a-b3e6-d358bf9dd00c" (UID: "61c076ae-c401-4b0a-b3e6-d358bf9dd00c"). InnerVolumeSpecName "kube-api-access-8lqq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:32.773317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.773299    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-dshm" pod "61c076ae-c401-4b0a-b3e6-d358bf9dd00c" (UID: "61c076ae-c401-4b0a-b3e6-d358bf9dd00c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:32.871244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.871143    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lqq6\" (UniqueName: \"kubernetes.io/projected/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-kube-api-access-8lqq6\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:32.871244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.871189    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:32.871244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.871203    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:32.871244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.871216    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:32.871244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.871228    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:32.871244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:32.871241    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c076ae-c401-4b0a-b3e6-d358bf9dd00c-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:33.357927 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:33.357897    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 30 23:56:33.457947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:33.457911    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4" event={"ID":"61c076ae-c401-4b0a-b3e6-d358bf9dd00c","Type":"ContainerDied","Data":"b92301ab7b49e7234452aba7f68ada5a94e3a9645db3504ac094b551fe5de1d2"}
Oct 30 23:56:33.458322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:33.458303    4555 scope.go:122] "RemoveContainer" containerID="369249b4a410e7b8f4215bf84ab464861f0f3d7296061ca591f4bea5c8ee9665"
Oct 30 23:56:33.458527 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:33.458408    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"
Oct 30 23:56:33.469241 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:33.469015    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4" podUID="61c076ae-c401-4b0a-b3e6-d358bf9dd00c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.146:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:56:33.508690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:33.508657    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"]
Oct 30 23:56:33.530187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:33.530157    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f026c1/test-model-f026c1-kserve-6675c47c8c-x9zx4"]
Oct 30 23:56:35.221876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:35.221843    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c076ae-c401-4b0a-b3e6-d358bf9dd00c" path="/var/lib/kubelet/pods/61c076ae-c401-4b0a-b3e6-d358bf9dd00c/volumes"
Oct 30 23:56:38.199880 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:38.199852    4555 scope.go:122] "RemoveContainer" containerID="523d6b1297853c12374e10b52ef5f4e63d8de65fda3f4eacd2ed410731fd3ef2"
Oct 30 23:56:39.517711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:39.517632    4555 generic.go:356] "Generic (PLEG): container finished" podID="48a90375-8b99-4bc0-9afe-075ed088493a" containerID="1696f8df1fc7f2c79611f3b0879ea891cb29838ad6fd534c7ebe877a12690044" exitCode=7
Oct 30 23:56:39.517711 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:39.517688    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj" event={"ID":"48a90375-8b99-4bc0-9afe-075ed088493a","Type":"ContainerDied","Data":"1696f8df1fc7f2c79611f3b0879ea891cb29838ad6fd534c7ebe877a12690044"}
Oct 30 23:56:39.518182 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:39.517734    4555 scope.go:122] "RemoveContainer" containerID="523d6b1297853c12374e10b52ef5f4e63d8de65fda3f4eacd2ed410731fd3ef2"
Oct 30 23:56:39.518182 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:39.518162    4555 scope.go:122] "RemoveContainer" containerID="1696f8df1fc7f2c79611f3b0879ea891cb29838ad6fd534c7ebe877a12690044"
Oct 30 23:56:39.518639 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:56:39.518611    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-4hnnj_odh-ai-gateway-infra(48a90375-8b99-4bc0-9afe-075ed088493a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj" podUID="48a90375-8b99-4bc0-9afe-075ed088493a"
Oct 30 23:56:40.612839 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:40.612742    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"]
Oct 30 23:56:41.049800 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.049767    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:41.107014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.106979    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-xks-ca\") pod \"48a90375-8b99-4bc0-9afe-075ed088493a\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") "
Oct 30 23:56:41.107328 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.107310    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a90375-8b99-4bc0-9afe-075ed088493a-kube-api-access-qcfzc\" (UniqueName: \"kubernetes.io/projected/48a90375-8b99-4bc0-9afe-075ed088493a-kube-api-access-qcfzc\") pod \"48a90375-8b99-4bc0-9afe-075ed088493a\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") "
Oct 30 23:56:41.107804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.107384    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-openshift-ca\") pod \"48a90375-8b99-4bc0-9afe-075ed088493a\" (UID: \"48a90375-8b99-4bc0-9afe-075ed088493a\") "
Oct 30 23:56:41.108213 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.108185    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-openshift-ca" pod "48a90375-8b99-4bc0-9afe-075ed088493a" (UID: "48a90375-8b99-4bc0-9afe-075ed088493a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:41.116314 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.113018    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-xks-ca" pod "48a90375-8b99-4bc0-9afe-075ed088493a" (UID: "48a90375-8b99-4bc0-9afe-075ed088493a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:41.116964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.116931    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a90375-8b99-4bc0-9afe-075ed088493a-kube-api-access-qcfzc" pod "48a90375-8b99-4bc0-9afe-075ed088493a" (UID: "48a90375-8b99-4bc0-9afe-075ed088493a"). InnerVolumeSpecName "kube-api-access-qcfzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:41.210394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.210347    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcfzc\" (UniqueName: \"kubernetes.io/projected/48a90375-8b99-4bc0-9afe-075ed088493a-kube-api-access-qcfzc\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:41.210394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.210387    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:41.210394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.210400    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a90375-8b99-4bc0-9afe-075ed088493a-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:41.624892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.624859    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"
Oct 30 23:56:41.624892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.624876    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj" event={"ID":"48a90375-8b99-4bc0-9afe-075ed088493a","Type":"ContainerDied","Data":"e1a3a9080ea4d929a07435baa837dc69a5a6fc7d2157c60fa876d884771a898f"}
Oct 30 23:56:41.625489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.624952    4555 scope.go:122] "RemoveContainer" containerID="1696f8df1fc7f2c79611f3b0879ea891cb29838ad6fd534c7ebe877a12690044"
Oct 30 23:56:41.654273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.654245    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"]
Oct 30 23:56:41.728034 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:41.728004    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-4hnnj"]
Oct 30 23:56:43.212166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:43.212133    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a90375-8b99-4bc0-9afe-075ed088493a" path="/var/lib/kubelet/pods/48a90375-8b99-4bc0-9afe-075ed088493a/volumes"
Oct 30 23:56:45.596598 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:45.596537    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl" podUID="0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" containerName="main" containerID="cri-o://9edb46cbf5cf5cd35959db61f6ef0c03d5619455bb888d7bd75846ba1ad25a4d" gracePeriod=45
Oct 30 23:56:45.821916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:45.821877    4555 generic.go:356] "Generic (PLEG): container finished" podID="0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" containerID="9edb46cbf5cf5cd35959db61f6ef0c03d5619455bb888d7bd75846ba1ad25a4d" exitCode=0
Oct 30 23:56:45.822090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:45.821950    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl" event={"ID":"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1","Type":"ContainerDied","Data":"9edb46cbf5cf5cd35959db61f6ef0c03d5619455bb888d7bd75846ba1ad25a4d"}
Oct 30 23:56:45.938599 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:45.938574    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:56:46.021545 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.021511    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tls-certs\") pod \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") "
Oct 30 23:56:46.021748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.021567    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tmp-dir\") pod \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") "
Oct 30 23:56:46.021748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.021704    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-dshm\") pod \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") "
Oct 30 23:56:46.021886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.021778    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-model-cache\") pod \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") "
Oct 30 23:56:46.021886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.021808    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-home\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-home\") pod \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") "
Oct 30 23:56:46.021886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.021867    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-kube-api-access-w9xn9\" (UniqueName: \"kubernetes.io/projected/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-kube-api-access-w9xn9\") pod \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\" (UID: \"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1\") "
Oct 30 23:56:46.022669 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.022571    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-model-cache" pod "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" (UID: "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:46.023032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.022673    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-home" pod "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" (UID: "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:46.023443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.023413    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tmp-dir" pod "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" (UID: "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:46.025846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.025819    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tls-certs" pod "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" (UID: "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:46.025980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.025957    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-kube-api-access-w9xn9" pod "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" (UID: "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1"). InnerVolumeSpecName "kube-api-access-w9xn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:46.026735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.026712    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-dshm" pod "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" (UID: "0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:46.124351 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.124317    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:46.124351 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.124349    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9xn9\" (UniqueName: \"kubernetes.io/projected/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-kube-api-access-w9xn9\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:46.124351 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.124360    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:46.124624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.124388    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:46.124624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.124399    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:46.124624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.124408    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:46.832341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.832301    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl" event={"ID":"0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1","Type":"ContainerDied","Data":"2afca62b711331619d0ebefdb62ccb9408084d7e9cd5120540ae6ffb46aead1c"}
Oct 30 23:56:46.832341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.832349    4555 scope.go:122] "RemoveContainer" containerID="9edb46cbf5cf5cd35959db61f6ef0c03d5619455bb888d7bd75846ba1ad25a4d"
Oct 30 23:56:46.832909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.832319    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"
Oct 30 23:56:46.861841 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.861809    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"]
Oct 30 23:56:46.866565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:46.866542    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-82d449/test-model-82d449-kserve-5498db5685-tbkhl"]
Oct 30 23:56:47.211690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:47.211612    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" path="/var/lib/kubelet/pods/0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1/volumes"
Oct 30 23:56:50.783136 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.783097    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"]
Oct 30 23:56:50.784335 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784312    4555 cpu_manager.go:414] "containerMap: removing container" podUID="61c076ae-c401-4b0a-b3e6-d358bf9dd00c" containerName="main"
Oct 30 23:56:50.784335 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784339    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c076ae-c401-4b0a-b3e6-d358bf9dd00c" containerName="main"
Oct 30 23:56:50.784464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784382    4555 cpu_manager.go:414] "containerMap: removing container" podUID="48a90375-8b99-4bc0-9afe-075ed088493a" containerName="revoke-keys"
Oct 30 23:56:50.784464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784391    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a90375-8b99-4bc0-9afe-075ed088493a" containerName="revoke-keys"
Oct 30 23:56:50.784464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784405    4555 cpu_manager.go:414] "containerMap: removing container" podUID="48a90375-8b99-4bc0-9afe-075ed088493a" containerName="revoke-keys"
Oct 30 23:56:50.784464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784414    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a90375-8b99-4bc0-9afe-075ed088493a" containerName="revoke-keys"
Oct 30 23:56:50.784603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784467    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" containerName="main"
Oct 30 23:56:50.784603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784474    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" containerName="main"
Oct 30 23:56:50.784692 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784680    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0ac0c9-f4d4-46d4-81b5-b702bff7f7d1" containerName="main"
Oct 30 23:56:50.784731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784704    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a90375-8b99-4bc0-9afe-075ed088493a" containerName="revoke-keys"
Oct 30 23:56:50.784731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784716    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c076ae-c401-4b0a-b3e6-d358bf9dd00c" containerName="main"
Oct 30 23:56:50.784731 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784728    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a90375-8b99-4bc0-9afe-075ed088493a" containerName="revoke-keys"
Oct 30 23:56:50.784833 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.784737    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a90375-8b99-4bc0-9afe-075ed088493a" containerName="revoke-keys"
Oct 30 23:56:50.794828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.794802    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:50.798354 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.798328    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"]
Oct 30 23:56:50.815702 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.815674    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:56:50.815922 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.815904    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:56:50.889342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.889312    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc4rr\" (UniqueName: \"kubernetes.io/projected/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-kube-api-access-cc4rr\") pod \"maas-api-revoke-sub-91354e50-rk8vq\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:50.889610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.889591    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-rk8vq\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:50.889928 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.889912    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-rk8vq\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:50.991546 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.991505    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-rk8vq\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:50.991759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.991745    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc4rr\" (UniqueName: \"kubernetes.io/projected/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-kube-api-access-cc4rr\") pod \"maas-api-revoke-sub-91354e50-rk8vq\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:50.991834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.991783    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-rk8vq\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:50.992715 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.992682    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-rk8vq\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:50.994319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:50.994287    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-rk8vq\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:51.007203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:51.007175    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc4rr\" (UniqueName: \"kubernetes.io/projected/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-kube-api-access-cc4rr\") pod \"maas-api-revoke-sub-91354e50-rk8vq\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:51.135443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:51.132571    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:56:51.541329 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:56:51.541297    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4fd746_c7ec_46c4_aaec_d685bd17e8b4.slice/crio-d63c1907c52685c8c639d8400cff54197c79dcc2b455c7f10e700dc900b17c2b WatchSource:0}: Error finding container d63c1907c52685c8c639d8400cff54197c79dcc2b455c7f10e700dc900b17c2b: Status 404 returned error can't find the container with id d63c1907c52685c8c639d8400cff54197c79dcc2b455c7f10e700dc900b17c2b
Oct 30 23:56:51.544762 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:51.544733    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"]
Oct 30 23:56:51.896954 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:51.896906    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq" event={"ID":"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4","Type":"ContainerStarted","Data":"d63c1907c52685c8c639d8400cff54197c79dcc2b455c7f10e700dc900b17c2b"}
Oct 30 23:56:52.908686 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:52.908590    4555 generic.go:356] "Generic (PLEG): container finished" podID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerID="4452507a8203c22b83cdb856e0037214247384a708620ac87049e43e12e4feef" exitCode=7
Oct 30 23:56:52.909187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:52.908677    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq" event={"ID":"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4","Type":"ContainerDied","Data":"4452507a8203c22b83cdb856e0037214247384a708620ac87049e43e12e4feef"}
Oct 30 23:56:52.909187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:52.909054    4555 scope.go:122] "RemoveContainer" containerID="4452507a8203c22b83cdb856e0037214247384a708620ac87049e43e12e4feef"
Oct 30 23:56:53.919085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:53.919046    4555 generic.go:356] "Generic (PLEG): container finished" podID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerID="ff515fdc12904a88e335b7e187a09d3a9a5e62cf1bf64940515d9d4aa455c642" exitCode=7
Oct 30 23:56:53.919559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:53.919117    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq" event={"ID":"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4","Type":"ContainerDied","Data":"ff515fdc12904a88e335b7e187a09d3a9a5e62cf1bf64940515d9d4aa455c642"}
Oct 30 23:56:53.919559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:53.919175    4555 scope.go:122] "RemoveContainer" containerID="4452507a8203c22b83cdb856e0037214247384a708620ac87049e43e12e4feef"
Oct 30 23:56:53.919659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:53.919565    4555 scope.go:122] "RemoveContainer" containerID="ff515fdc12904a88e335b7e187a09d3a9a5e62cf1bf64940515d9d4aa455c642"
Oct 30 23:56:53.920016 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:56:53.919995    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-rk8vq_odh-ai-gateway-infra(7b4fd746-c7ec-46c4-aaec-d685bd17e8b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4"
Oct 30 23:56:54.930536 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:54.930505    4555 scope.go:122] "RemoveContainer" containerID="ff515fdc12904a88e335b7e187a09d3a9a5e62cf1bf64940515d9d4aa455c642"
Oct 30 23:56:54.930948 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:56:54.930861    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-rk8vq_odh-ai-gateway-infra(7b4fd746-c7ec-46c4-aaec-d685bd17e8b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4"
Oct 30 23:56:56.734390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:56:56.734343    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:56:57.462436 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:56:57.462350    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:57:05.200792 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:05.200763    4555 scope.go:122] "RemoveContainer" containerID="ff515fdc12904a88e335b7e187a09d3a9a5e62cf1bf64940515d9d4aa455c642"
Oct 30 23:57:06.238631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:06.238599    4555 generic.go:356] "Generic (PLEG): container finished" podID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerID="13c1ff36805a25f70967bcd6fd6e8d9074db6e8ff4a6bee71b835b8940f6431c" exitCode=7
Oct 30 23:57:06.239098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:06.238673    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq" event={"ID":"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4","Type":"ContainerDied","Data":"13c1ff36805a25f70967bcd6fd6e8d9074db6e8ff4a6bee71b835b8940f6431c"}
Oct 30 23:57:06.239098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:06.238745    4555 scope.go:122] "RemoveContainer" containerID="ff515fdc12904a88e335b7e187a09d3a9a5e62cf1bf64940515d9d4aa455c642"
Oct 30 23:57:06.239236 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:06.239220    4555 scope.go:122] "RemoveContainer" containerID="13c1ff36805a25f70967bcd6fd6e8d9074db6e8ff4a6bee71b835b8940f6431c"
Oct 30 23:57:06.239741 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:06.239714    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-rk8vq_odh-ai-gateway-infra(7b4fd746-c7ec-46c4-aaec-d685bd17e8b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4"
Oct 30 23:57:07.278516 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.278480    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"]
Oct 30 23:57:07.479318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.479295    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:57:07.637091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.636987    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-openshift-ca\") pod \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") "
Oct 30 23:57:07.637091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.637067    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-xks-ca\") pod \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") "
Oct 30 23:57:07.637336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.637309    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-kube-api-access-cc4rr\" (UniqueName: \"kubernetes.io/projected/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-kube-api-access-cc4rr\") pod \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\" (UID: \"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4\") "
Oct 30 23:57:07.637459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.637425    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-openshift-ca" pod "7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" (UID: "7b4fd746-c7ec-46c4-aaec-d685bd17e8b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:07.638184 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.638160    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:07.639339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.639316    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-xks-ca" pod "7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" (UID: "7b4fd746-c7ec-46c4-aaec-d685bd17e8b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:07.639564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.639547    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-kube-api-access-cc4rr" pod "7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" (UID: "7b4fd746-c7ec-46c4-aaec-d685bd17e8b4"). InnerVolumeSpecName "kube-api-access-cc4rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:07.740113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.740076    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:07.740113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:07.740115    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc4rr\" (UniqueName: \"kubernetes.io/projected/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4-kube-api-access-cc4rr\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:08.263877 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:08.263845    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"
Oct 30 23:57:08.264110 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:08.263848    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq" event={"ID":"7b4fd746-c7ec-46c4-aaec-d685bd17e8b4","Type":"ContainerDied","Data":"d63c1907c52685c8c639d8400cff54197c79dcc2b455c7f10e700dc900b17c2b"}
Oct 30 23:57:08.264110 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:08.263981    4555 scope.go:122] "RemoveContainer" containerID="13c1ff36805a25f70967bcd6fd6e8d9074db6e8ff4a6bee71b835b8940f6431c"
Oct 30 23:57:08.299101 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:08.299064    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"]
Oct 30 23:57:08.305308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:08.305280    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-rk8vq"]
Oct 30 23:57:09.210291 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:09.210260    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" path="/var/lib/kubelet/pods/7b4fd746-c7ec-46c4-aaec-d685bd17e8b4/volumes"
Oct 30 23:57:11.937835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.937760    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"]
Oct 30 23:57:11.938640 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.938620    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerName="revoke-keys"
Oct 30 23:57:11.938696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.938641    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerName="revoke-keys"
Oct 30 23:57:11.938696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.938663    4555 cpu_manager.go:414] "containerMap: removing container" podUID="48a90375-8b99-4bc0-9afe-075ed088493a" containerName="revoke-keys"
Oct 30 23:57:11.938696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.938668    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a90375-8b99-4bc0-9afe-075ed088493a" containerName="revoke-keys"
Oct 30 23:57:11.938696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.938687    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerName="revoke-keys"
Oct 30 23:57:11.938696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.938691    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerName="revoke-keys"
Oct 30 23:57:11.938848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.938822    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerName="revoke-keys"
Oct 30 23:57:11.938848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.938833    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerName="revoke-keys"
Oct 30 23:57:11.943871 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.943855    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:11.949317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.949276    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:57:11.951894 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.951874    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:57:11.964389 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:11.953789    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"]
Oct 30 23:57:12.050591 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.027196    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44162d75-b4ff-4620-9257-941c78292d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-jkqrt\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:12.050890 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.050864    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44162d75-b4ff-4620-9257-941c78292d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-jkqrt\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:12.051230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.051212    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp6qn\" (UniqueName: \"kubernetes.io/projected/44162d75-b4ff-4620-9257-941c78292d69-kube-api-access-gp6qn\") pod \"maas-api-revoke-sub-91354e50-jkqrt\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:12.156304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.156276    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp6qn\" (UniqueName: \"kubernetes.io/projected/44162d75-b4ff-4620-9257-941c78292d69-kube-api-access-gp6qn\") pod \"maas-api-revoke-sub-91354e50-jkqrt\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:12.156571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.156553    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44162d75-b4ff-4620-9257-941c78292d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-jkqrt\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:12.156636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.156583    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44162d75-b4ff-4620-9257-941c78292d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-jkqrt\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:12.157560 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.157340    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44162d75-b4ff-4620-9257-941c78292d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-jkqrt\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:12.159316 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.159285    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44162d75-b4ff-4620-9257-941c78292d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-jkqrt\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:12.178430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.178357    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp6qn\" (UniqueName: \"kubernetes.io/projected/44162d75-b4ff-4620-9257-941c78292d69-kube-api-access-gp6qn\") pod \"maas-api-revoke-sub-91354e50-jkqrt\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:12.375486 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.375458    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:12.670822 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:57:12.670792    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44162d75_b4ff_4620_9257_941c78292d69.slice/crio-506641e36f8634ab174ef59afdeda05f37dec02ce199bf57098e8ff2682ce02a WatchSource:0}: Error finding container 506641e36f8634ab174ef59afdeda05f37dec02ce199bf57098e8ff2682ce02a: Status 404 returned error can't find the container with id 506641e36f8634ab174ef59afdeda05f37dec02ce199bf57098e8ff2682ce02a
Oct 30 23:57:12.672511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:12.672482    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"]
Oct 30 23:57:13.330947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:13.330908    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" event={"ID":"44162d75-b4ff-4620-9257-941c78292d69","Type":"ContainerStarted","Data":"21ab7286ac5c55e8edb64a7ae726e977af73f235ccce726569cc5d60ed5a825e"}
Oct 30 23:57:13.331358 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:13.330964    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" event={"ID":"44162d75-b4ff-4620-9257-941c78292d69","Type":"ContainerStarted","Data":"506641e36f8634ab174ef59afdeda05f37dec02ce199bf57098e8ff2682ce02a"}
Oct 30 23:57:13.355569 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:13.355518    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" podStartSLOduration=2.355503759 podStartE2EDuration="2.355503759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:13.35005995 +0000 UTC m=+2292.276697563" watchObservedRunningTime="2026-10-30 23:57:13.355503759 +0000 UTC m=+2292.282141339"
Oct 30 23:57:14.341505 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:14.341472    4555 generic.go:356] "Generic (PLEG): container finished" podID="44162d75-b4ff-4620-9257-941c78292d69" containerID="21ab7286ac5c55e8edb64a7ae726e977af73f235ccce726569cc5d60ed5a825e" exitCode=7
Oct 30 23:57:14.341927 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:14.341568    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" event={"ID":"44162d75-b4ff-4620-9257-941c78292d69","Type":"ContainerDied","Data":"21ab7286ac5c55e8edb64a7ae726e977af73f235ccce726569cc5d60ed5a825e"}
Oct 30 23:57:14.341927 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:14.341877    4555 scope.go:122] "RemoveContainer" containerID="21ab7286ac5c55e8edb64a7ae726e977af73f235ccce726569cc5d60ed5a825e"
Oct 30 23:57:15.352567 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:15.352531    4555 generic.go:356] "Generic (PLEG): container finished" podID="44162d75-b4ff-4620-9257-941c78292d69" containerID="5bf8cb3890c60ae3277e0bf6ea5be61a314e931b67b8b46e5e61af9824ab26f6" exitCode=7
Oct 30 23:57:15.352978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:15.352597    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" event={"ID":"44162d75-b4ff-4620-9257-941c78292d69","Type":"ContainerDied","Data":"5bf8cb3890c60ae3277e0bf6ea5be61a314e931b67b8b46e5e61af9824ab26f6"}
Oct 30 23:57:15.352978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:15.352664    4555 scope.go:122] "RemoveContainer" containerID="21ab7286ac5c55e8edb64a7ae726e977af73f235ccce726569cc5d60ed5a825e"
Oct 30 23:57:15.353087 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:15.353032    4555 scope.go:122] "RemoveContainer" containerID="5bf8cb3890c60ae3277e0bf6ea5be61a314e931b67b8b46e5e61af9824ab26f6"
Oct 30 23:57:15.353534 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:15.353512    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-jkqrt_odh-ai-gateway-infra(44162d75-b4ff-4620-9257-941c78292d69)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" podUID="44162d75-b4ff-4620-9257-941c78292d69"
Oct 30 23:57:16.363408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:16.363364    4555 scope.go:122] "RemoveContainer" containerID="5bf8cb3890c60ae3277e0bf6ea5be61a314e931b67b8b46e5e61af9824ab26f6"
Oct 30 23:57:16.363827 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:16.363715    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-jkqrt_odh-ai-gateway-infra(44162d75-b4ff-4620-9257-941c78292d69)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" podUID="44162d75-b4ff-4620-9257-941c78292d69"
Oct 30 23:57:18.107039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:18.106902    4555 ???:1] "http: TLS handshake error from 10.0.137.81:39779: EOF"
Oct 30 23:57:18.107039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:18.106994    4555 ???:1] "http: TLS handshake error from 10.0.137.81:57583: EOF"
Oct 30 23:57:18.107636 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:18.107147    4555 ???:1] "http: TLS handshake error from 10.0.137.81:55983: EOF"
Oct 30 23:57:23.827301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.827269    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"]
Oct 30 23:57:23.828066 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.828050    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerName="revoke-keys"
Oct 30 23:57:23.828114 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.828067    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerName="revoke-keys"
Oct 30 23:57:23.828232 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.828223    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4fd746-c7ec-46c4-aaec-d685bd17e8b4" containerName="revoke-keys"
Oct 30 23:57:23.833673 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.833655    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:23.843475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.843448    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"]
Oct 30 23:57:23.876923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.876885    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:23.877123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.876960    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sllcm\" (UniqueName: \"kubernetes.io/projected/789133d1-763e-4f3f-aec7-72a5810a642b-kube-api-access-sllcm\") pod \"maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:23.877476 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.877448    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:23.979099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.979069    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:23.979283 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.979122    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sllcm\" (UniqueName: \"kubernetes.io/projected/789133d1-763e-4f3f-aec7-72a5810a642b-kube-api-access-sllcm\") pod \"maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:23.979475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.979455    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:23.979953 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.979931    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:23.982038 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.982013    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:23.992064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:23.992043    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sllcm\" (UniqueName: \"kubernetes.io/projected/789133d1-763e-4f3f-aec7-72a5810a642b-kube-api-access-sllcm\") pod \"maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:24.154305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:24.154229    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:24.452224 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:57:24.452187    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789133d1_763e_4f3f_aec7_72a5810a642b.slice/crio-13e0627fe5eaf1830ab6911426d53dde71ea7b0afd9beef0d9ef9c543f438bff WatchSource:0}: Error finding container 13e0627fe5eaf1830ab6911426d53dde71ea7b0afd9beef0d9ef9c543f438bff: Status 404 returned error can't find the container with id 13e0627fe5eaf1830ab6911426d53dde71ea7b0afd9beef0d9ef9c543f438bff
Oct 30 23:57:24.453019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:24.452984    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"]
Oct 30 23:57:24.551148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:24.551116    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm" event={"ID":"789133d1-763e-4f3f-aec7-72a5810a642b","Type":"ContainerStarted","Data":"13e0627fe5eaf1830ab6911426d53dde71ea7b0afd9beef0d9ef9c543f438bff"}
Oct 30 23:57:25.562103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:25.562059    4555 generic.go:356] "Generic (PLEG): container finished" podID="789133d1-763e-4f3f-aec7-72a5810a642b" containerID="cdc9ffa2b0e1b7ee79d0f19288f0f59e21ef1619df4ebba05a82461f6b459f49" exitCode=0
Oct 30 23:57:25.562567 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:25.562195    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm" event={"ID":"789133d1-763e-4f3f-aec7-72a5810a642b","Type":"ContainerDied","Data":"cdc9ffa2b0e1b7ee79d0f19288f0f59e21ef1619df4ebba05a82461f6b459f49"}
Oct 30 23:57:26.767137 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:26.767110    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:26.845063 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:26.844975    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789133d1-763e-4f3f-aec7-72a5810a642b-kube-api-access-sllcm\" (UniqueName: \"kubernetes.io/projected/789133d1-763e-4f3f-aec7-72a5810a642b-kube-api-access-sllcm\") pod \"789133d1-763e-4f3f-aec7-72a5810a642b\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") "
Oct 30 23:57:26.845247 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:26.845073    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-openshift-ca\") pod \"789133d1-763e-4f3f-aec7-72a5810a642b\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") "
Oct 30 23:57:26.845350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:26.845335    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-xks-ca\") pod \"789133d1-763e-4f3f-aec7-72a5810a642b\" (UID: \"789133d1-763e-4f3f-aec7-72a5810a642b\") "
Oct 30 23:57:26.845600 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:26.845530    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-openshift-ca" pod "789133d1-763e-4f3f-aec7-72a5810a642b" (UID: "789133d1-763e-4f3f-aec7-72a5810a642b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:26.846516 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:26.846492    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:26.847701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:26.847666    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-xks-ca" pod "789133d1-763e-4f3f-aec7-72a5810a642b" (UID: "789133d1-763e-4f3f-aec7-72a5810a642b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:26.847812 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:26.847761    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789133d1-763e-4f3f-aec7-72a5810a642b-kube-api-access-sllcm" pod "789133d1-763e-4f3f-aec7-72a5810a642b" (UID: "789133d1-763e-4f3f-aec7-72a5810a642b"). InnerVolumeSpecName "kube-api-access-sllcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:26.948098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:26.948063    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sllcm\" (UniqueName: \"kubernetes.io/projected/789133d1-763e-4f3f-aec7-72a5810a642b-kube-api-access-sllcm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:26.948098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:26.948094    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789133d1-763e-4f3f-aec7-72a5810a642b-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:27.448698 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:27.448656    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:57:27.584970 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:27.584938    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"
Oct 30 23:57:27.585147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:27.584964    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm" event={"ID":"789133d1-763e-4f3f-aec7-72a5810a642b","Type":"ContainerDied","Data":"13e0627fe5eaf1830ab6911426d53dde71ea7b0afd9beef0d9ef9c543f438bff"}
Oct 30 23:57:27.585147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:27.585032    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13e0627fe5eaf1830ab6911426d53dde71ea7b0afd9beef0d9ef9c543f438bff"
Oct 30 23:57:28.200215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.200177    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"]
Oct 30 23:57:28.201451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.201428    4555 cpu_manager.go:414] "containerMap: removing container" podUID="789133d1-763e-4f3f-aec7-72a5810a642b" containerName="revoke-keys"
Oct 30 23:57:28.201551 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.201453    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789133d1-763e-4f3f-aec7-72a5810a642b" containerName="revoke-keys"
Oct 30 23:57:28.201724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.201711    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="789133d1-763e-4f3f-aec7-72a5810a642b" containerName="revoke-keys"
Oct 30 23:57:28.206467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.206446    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.207593 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.207479    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"]
Oct 30 23:57:28.219227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.219201    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"]
Oct 30 23:57:28.221196 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.221169    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.224751 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.224718    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"]
Oct 30 23:57:28.272045 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.272012    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7tq6\" (UniqueName: \"kubernetes.io/projected/04ec3407-e40d-4e3e-8777-737251989527-kube-api-access-c7tq6\") pod \"maas-api-revoke-sub-92c9c9fc-46mp9\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.272227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.272062    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xg52\" (UniqueName: \"kubernetes.io/projected/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-kube-api-access-6xg52\") pod \"maas-api-revoke-sub-f993bcbd-477cz\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.272227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.272211    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ec3407-e40d-4e3e-8777-737251989527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c9c9fc-46mp9\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.272346 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.272284    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ec3407-e40d-4e3e-8777-737251989527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c9c9fc-46mp9\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.272454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.272345    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-477cz\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.272532 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.272464    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-477cz\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.374735 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.374696    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7tq6\" (UniqueName: \"kubernetes.io/projected/04ec3407-e40d-4e3e-8777-737251989527-kube-api-access-c7tq6\") pod \"maas-api-revoke-sub-92c9c9fc-46mp9\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.374939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.374766    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xg52\" (UniqueName: \"kubernetes.io/projected/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-kube-api-access-6xg52\") pod \"maas-api-revoke-sub-f993bcbd-477cz\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.374939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.374924    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ec3407-e40d-4e3e-8777-737251989527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c9c9fc-46mp9\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.375055 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.375007    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ec3407-e40d-4e3e-8777-737251989527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c9c9fc-46mp9\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.375097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.375071    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-477cz\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.375245 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.375229    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-477cz\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.375829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.375800    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ec3407-e40d-4e3e-8777-737251989527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c9c9fc-46mp9\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.375949 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.375931    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-477cz\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.377923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.377900    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-477cz\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.378210 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.378182    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ec3407-e40d-4e3e-8777-737251989527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c9c9fc-46mp9\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.388979 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.388952    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7tq6\" (UniqueName: \"kubernetes.io/projected/04ec3407-e40d-4e3e-8777-737251989527-kube-api-access-c7tq6\") pod \"maas-api-revoke-sub-92c9c9fc-46mp9\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.388979 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.388970    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xg52\" (UniqueName: \"kubernetes.io/projected/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-kube-api-access-6xg52\") pod \"maas-api-revoke-sub-f993bcbd-477cz\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.550264 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.550217    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:28.556842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.556815    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:28.952180 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.950605    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"]
Oct 30 23:57:28.955016 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:57:28.953160    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ec3407_e40d_4e3e_8777_737251989527.slice/crio-20fd24a46b31556bf8707c2046c98e97bd117c543a98b64e772855e307181b84 WatchSource:0}: Error finding container 20fd24a46b31556bf8707c2046c98e97bd117c543a98b64e772855e307181b84: Status 404 returned error can't find the container with id 20fd24a46b31556bf8707c2046c98e97bd117c543a98b64e772855e307181b84
Oct 30 23:57:28.978075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:28.978047    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"]
Oct 30 23:57:28.980208 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:57:28.980165    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8ae5a9_9c20_4b55_a8cb_bde2e21057ec.slice/crio-ff34fa623d864428dfd5b7c4f2410d3c222e80eabeb2fa798b036c5c078872a1 WatchSource:0}: Error finding container ff34fa623d864428dfd5b7c4f2410d3c222e80eabeb2fa798b036c5c078872a1: Status 404 returned error can't find the container with id ff34fa623d864428dfd5b7c4f2410d3c222e80eabeb2fa798b036c5c078872a1
Oct 30 23:57:29.200240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:29.200153    4555 scope.go:122] "RemoveContainer" containerID="5bf8cb3890c60ae3277e0bf6ea5be61a314e931b67b8b46e5e61af9824ab26f6"
Oct 30 23:57:29.611251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:29.611218    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" event={"ID":"44162d75-b4ff-4620-9257-941c78292d69","Type":"ContainerStarted","Data":"0abce34f7aacbd80045537f264224c257bc8936a64d9d823da1d38cda5218ad2"}
Oct 30 23:57:29.613407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:29.613361    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9" event={"ID":"04ec3407-e40d-4e3e-8777-737251989527","Type":"ContainerStarted","Data":"d48f9a5fc140ca0c14d9dab9d3fa2492f8b38c5c47bd5fa1cdff3e58d6961e65"}
Oct 30 23:57:29.613559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:29.613429    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9" event={"ID":"04ec3407-e40d-4e3e-8777-737251989527","Type":"ContainerStarted","Data":"20fd24a46b31556bf8707c2046c98e97bd117c543a98b64e772855e307181b84"}
Oct 30 23:57:29.615142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:29.615118    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" event={"ID":"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec","Type":"ContainerStarted","Data":"cef91cc7985f17f8bbf2183f51052ac4746c25c162d51ee593e47438e35fd35a"}
Oct 30 23:57:29.615258 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:29.615161    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" event={"ID":"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec","Type":"ContainerStarted","Data":"ff34fa623d864428dfd5b7c4f2410d3c222e80eabeb2fa798b036c5c078872a1"}
Oct 30 23:57:29.702067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:29.699003    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9" podStartSLOduration=1.698975149 podStartE2EDuration="1.698975149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:29.672090768 +0000 UTC m=+2308.598728356" watchObservedRunningTime="2026-10-30 23:57:29.698975149 +0000 UTC m=+2308.625612739"
Oct 30 23:57:29.727454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:29.724359    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" podStartSLOduration=1.724334642 podStartE2EDuration="1.724334642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:29.723180645 +0000 UTC m=+2308.649818233" watchObservedRunningTime="2026-10-30 23:57:29.724334642 +0000 UTC m=+2308.650972229"
Oct 30 23:57:30.627247 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:30.627208    4555 generic.go:356] "Generic (PLEG): container finished" podID="44162d75-b4ff-4620-9257-941c78292d69" containerID="0abce34f7aacbd80045537f264224c257bc8936a64d9d823da1d38cda5218ad2" exitCode=7
Oct 30 23:57:30.627798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:30.627248    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" event={"ID":"44162d75-b4ff-4620-9257-941c78292d69","Type":"ContainerDied","Data":"0abce34f7aacbd80045537f264224c257bc8936a64d9d823da1d38cda5218ad2"}
Oct 30 23:57:30.627798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:30.627322    4555 scope.go:122] "RemoveContainer" containerID="5bf8cb3890c60ae3277e0bf6ea5be61a314e931b67b8b46e5e61af9824ab26f6"
Oct 30 23:57:30.627892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:30.627861    4555 scope.go:122] "RemoveContainer" containerID="0abce34f7aacbd80045537f264224c257bc8936a64d9d823da1d38cda5218ad2"
Oct 30 23:57:30.628350 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:30.628329    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-jkqrt_odh-ai-gateway-infra(44162d75-b4ff-4620-9257-941c78292d69)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" podUID="44162d75-b4ff-4620-9257-941c78292d69"
Oct 30 23:57:30.630785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:30.630760    4555 generic.go:356] "Generic (PLEG): container finished" podID="04ec3407-e40d-4e3e-8777-737251989527" containerID="d48f9a5fc140ca0c14d9dab9d3fa2492f8b38c5c47bd5fa1cdff3e58d6961e65" exitCode=0
Oct 30 23:57:30.630974 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:30.630852    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9" event={"ID":"04ec3407-e40d-4e3e-8777-737251989527","Type":"ContainerDied","Data":"d48f9a5fc140ca0c14d9dab9d3fa2492f8b38c5c47bd5fa1cdff3e58d6961e65"}
Oct 30 23:57:30.634088 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:30.634066    4555 generic.go:356] "Generic (PLEG): container finished" podID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerID="cef91cc7985f17f8bbf2183f51052ac4746c25c162d51ee593e47438e35fd35a" exitCode=7
Oct 30 23:57:30.634224 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:30.634157    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" event={"ID":"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec","Type":"ContainerDied","Data":"cef91cc7985f17f8bbf2183f51052ac4746c25c162d51ee593e47438e35fd35a"}
Oct 30 23:57:30.634632 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:30.634612    4555 scope.go:122] "RemoveContainer" containerID="cef91cc7985f17f8bbf2183f51052ac4746c25c162d51ee593e47438e35fd35a"
Oct 30 23:57:30.690846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:30.690562    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"]
Oct 30 23:57:31.646818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:31.646781    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" event={"ID":"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec","Type":"ContainerStarted","Data":"6e66fe8d38cdc44ea570e55cb36e8f646a597306d12ffe8b45e4a868d3b330da"}
Oct 30 23:57:31.886494 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:31.885750    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:31.956433 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:31.956400    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:31.980239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:31.980178    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44162d75-b4ff-4620-9257-941c78292d69-kube-api-access-gp6qn\" (UniqueName: \"kubernetes.io/projected/44162d75-b4ff-4620-9257-941c78292d69-kube-api-access-gp6qn\") pod \"44162d75-b4ff-4620-9257-941c78292d69\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") "
Oct 30 23:57:31.980448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:31.980335    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44162d75-b4ff-4620-9257-941c78292d69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44162d75-b4ff-4620-9257-941c78292d69-maas-api-xks-ca\") pod \"44162d75-b4ff-4620-9257-941c78292d69\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") "
Oct 30 23:57:31.980690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:31.980663    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44162d75-b4ff-4620-9257-941c78292d69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44162d75-b4ff-4620-9257-941c78292d69-maas-api-openshift-ca\") pod \"44162d75-b4ff-4620-9257-941c78292d69\" (UID: \"44162d75-b4ff-4620-9257-941c78292d69\") "
Oct 30 23:57:31.982583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:31.982553    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44162d75-b4ff-4620-9257-941c78292d69-maas-api-openshift-ca" pod "44162d75-b4ff-4620-9257-941c78292d69" (UID: "44162d75-b4ff-4620-9257-941c78292d69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:31.983269 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:31.983238    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44162d75-b4ff-4620-9257-941c78292d69-kube-api-access-gp6qn" pod "44162d75-b4ff-4620-9257-941c78292d69" (UID: "44162d75-b4ff-4620-9257-941c78292d69"). InnerVolumeSpecName "kube-api-access-gp6qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:31.990053 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:31.990023    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44162d75-b4ff-4620-9257-941c78292d69-maas-api-xks-ca" pod "44162d75-b4ff-4620-9257-941c78292d69" (UID: "44162d75-b4ff-4620-9257-941c78292d69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:32.095576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.095474    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ec3407-e40d-4e3e-8777-737251989527-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ec3407-e40d-4e3e-8777-737251989527-maas-api-openshift-ca\") pod \"04ec3407-e40d-4e3e-8777-737251989527\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") "
Oct 30 23:57:32.096670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.096436    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ec3407-e40d-4e3e-8777-737251989527-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ec3407-e40d-4e3e-8777-737251989527-maas-api-xks-ca\") pod \"04ec3407-e40d-4e3e-8777-737251989527\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") "
Oct 30 23:57:32.132107 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.132062    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ec3407-e40d-4e3e-8777-737251989527-kube-api-access-c7tq6\" (UniqueName: \"kubernetes.io/projected/04ec3407-e40d-4e3e-8777-737251989527-kube-api-access-c7tq6\") pod \"04ec3407-e40d-4e3e-8777-737251989527\" (UID: \"04ec3407-e40d-4e3e-8777-737251989527\") "
Oct 30 23:57:32.133001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.105413    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ec3407-e40d-4e3e-8777-737251989527-maas-api-openshift-ca" pod "04ec3407-e40d-4e3e-8777-737251989527" (UID: "04ec3407-e40d-4e3e-8777-737251989527"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:32.133001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.123528    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ec3407-e40d-4e3e-8777-737251989527-maas-api-xks-ca" pod "04ec3407-e40d-4e3e-8777-737251989527" (UID: "04ec3407-e40d-4e3e-8777-737251989527"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:32.141441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.133874    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44162d75-b4ff-4620-9257-941c78292d69-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:32.165001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.137488    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ec3407-e40d-4e3e-8777-737251989527-kube-api-access-c7tq6" pod "04ec3407-e40d-4e3e-8777-737251989527" (UID: "04ec3407-e40d-4e3e-8777-737251989527"). InnerVolumeSpecName "kube-api-access-c7tq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:32.165001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.164942    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ec3407-e40d-4e3e-8777-737251989527-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:32.165001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.164963    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44162d75-b4ff-4620-9257-941c78292d69-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:32.165001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.164979    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ec3407-e40d-4e3e-8777-737251989527-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:32.165001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.165002    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp6qn\" (UniqueName: \"kubernetes.io/projected/44162d75-b4ff-4620-9257-941c78292d69-kube-api-access-gp6qn\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:32.267031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.267001    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7tq6\" (UniqueName: \"kubernetes.io/projected/04ec3407-e40d-4e3e-8777-737251989527-kube-api-access-c7tq6\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:32.661145 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.661100    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt" event={"ID":"44162d75-b4ff-4620-9257-941c78292d69","Type":"ContainerDied","Data":"506641e36f8634ab174ef59afdeda05f37dec02ce199bf57098e8ff2682ce02a"}
Oct 30 23:57:32.661724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.661117    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"
Oct 30 23:57:32.661724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.661217    4555 scope.go:122] "RemoveContainer" containerID="0abce34f7aacbd80045537f264224c257bc8936a64d9d823da1d38cda5218ad2"
Oct 30 23:57:32.663355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.663187    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"
Oct 30 23:57:32.663355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.663208    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9" event={"ID":"04ec3407-e40d-4e3e-8777-737251989527","Type":"ContainerDied","Data":"20fd24a46b31556bf8707c2046c98e97bd117c543a98b64e772855e307181b84"}
Oct 30 23:57:32.663355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.663277    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20fd24a46b31556bf8707c2046c98e97bd117c543a98b64e772855e307181b84"
Oct 30 23:57:32.666030 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.665991    4555 generic.go:356] "Generic (PLEG): container finished" podID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerID="6e66fe8d38cdc44ea570e55cb36e8f646a597306d12ffe8b45e4a868d3b330da" exitCode=7
Oct 30 23:57:32.666172 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.666159    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" event={"ID":"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec","Type":"ContainerDied","Data":"6e66fe8d38cdc44ea570e55cb36e8f646a597306d12ffe8b45e4a868d3b330da"}
Oct 30 23:57:32.666614 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.666596    4555 scope.go:122] "RemoveContainer" containerID="6e66fe8d38cdc44ea570e55cb36e8f646a597306d12ffe8b45e4a868d3b330da"
Oct 30 23:57:32.666962 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:32.666941    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f993bcbd-477cz_odh-ai-gateway-infra(9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec"
Oct 30 23:57:32.677766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.677740    4555 scope.go:122] "RemoveContainer" containerID="cef91cc7985f17f8bbf2183f51052ac4746c25c162d51ee593e47438e35fd35a"
Oct 30 23:57:32.719340 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.719304    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"]
Oct 30 23:57:32.727628 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:32.727597    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-jkqrt"]
Oct 30 23:57:33.210626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:33.210589    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44162d75-b4ff-4620-9257-941c78292d69" path="/var/lib/kubelet/pods/44162d75-b4ff-4620-9257-941c78292d69/volumes"
Oct 30 23:57:33.686300 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:33.686272    4555 scope.go:122] "RemoveContainer" containerID="6e66fe8d38cdc44ea570e55cb36e8f646a597306d12ffe8b45e4a868d3b330da"
Oct 30 23:57:33.686798 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:33.686756    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f993bcbd-477cz_odh-ai-gateway-infra(9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec"
Oct 30 23:57:41.978986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.978954    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"]
Oct 30 23:57:41.979777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979761    4555 cpu_manager.go:414] "containerMap: removing container" podUID="44162d75-b4ff-4620-9257-941c78292d69" containerName="revoke-keys"
Oct 30 23:57:41.979832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979779    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44162d75-b4ff-4620-9257-941c78292d69" containerName="revoke-keys"
Oct 30 23:57:41.979832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979796    4555 cpu_manager.go:414] "containerMap: removing container" podUID="04ec3407-e40d-4e3e-8777-737251989527" containerName="revoke-keys"
Oct 30 23:57:41.979832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979802    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ec3407-e40d-4e3e-8777-737251989527" containerName="revoke-keys"
Oct 30 23:57:41.979832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979825    4555 cpu_manager.go:414] "containerMap: removing container" podUID="44162d75-b4ff-4620-9257-941c78292d69" containerName="revoke-keys"
Oct 30 23:57:41.979832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979830    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44162d75-b4ff-4620-9257-941c78292d69" containerName="revoke-keys"
Oct 30 23:57:41.979986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979838    4555 cpu_manager.go:414] "containerMap: removing container" podUID="44162d75-b4ff-4620-9257-941c78292d69" containerName="revoke-keys"
Oct 30 23:57:41.979986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979845    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44162d75-b4ff-4620-9257-941c78292d69" containerName="revoke-keys"
Oct 30 23:57:41.979986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979972    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="44162d75-b4ff-4620-9257-941c78292d69" containerName="revoke-keys"
Oct 30 23:57:41.979986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979982    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ec3407-e40d-4e3e-8777-737251989527" containerName="revoke-keys"
Oct 30 23:57:41.980106 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979991    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="44162d75-b4ff-4620-9257-941c78292d69" containerName="revoke-keys"
Oct 30 23:57:41.980106 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.979999    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="44162d75-b4ff-4620-9257-941c78292d69" containerName="revoke-keys"
Oct 30 23:57:41.985338 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.985321    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:41.997547 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:41.997516    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"]
Oct 30 23:57:42.099345 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.099311    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-2x69x\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:42.101118 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.101089    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-2x69x\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:42.101507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.101491    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrhf\" (UniqueName: \"kubernetes.io/projected/f7be6f05-8295-48e2-a176-a6ba4d22b085-kube-api-access-vmrhf\") pod \"maas-api-revoke-sub-91354e50-2x69x\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:42.203806 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.203776    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrhf\" (UniqueName: \"kubernetes.io/projected/f7be6f05-8295-48e2-a176-a6ba4d22b085-kube-api-access-vmrhf\") pod \"maas-api-revoke-sub-91354e50-2x69x\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:42.203982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.203940    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-2x69x\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:42.203982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.203977    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-2x69x\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:42.204642 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.204615    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-2x69x\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:42.206742 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.206714    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-2x69x\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:42.219475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.219455    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrhf\" (UniqueName: \"kubernetes.io/projected/f7be6f05-8295-48e2-a176-a6ba4d22b085-kube-api-access-vmrhf\") pod \"maas-api-revoke-sub-91354e50-2x69x\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:42.313496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.313461    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:57:42.641431 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.635260    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"]
Oct 30 23:57:42.946099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:42.946066    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x" event={"ID":"f7be6f05-8295-48e2-a176-a6ba4d22b085","Type":"ContainerStarted","Data":"2e079b6d546919773bbf5f8a2f4bbc3eff292bc68e12345f3f9ce1bf653f5584"}
Oct 30 23:57:43.425761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.425726    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"]
Oct 30 23:57:43.430809 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.430787    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.457740 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.457707    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"]
Oct 30 23:57:43.468083 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.467939    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.468083 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.467994    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljtgl\" (UniqueName: \"kubernetes.io/projected/a016483e-936f-45f5-a6c8-62bc13386af1-kube-api-access-ljtgl\") pod \"maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.468298 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.468247    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.596393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.589306    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.596393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.589388    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljtgl\" (UniqueName: \"kubernetes.io/projected/a016483e-936f-45f5-a6c8-62bc13386af1-kube-api-access-ljtgl\") pod \"maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.596393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.589658    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.596705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.596622    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.608393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.599874    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.622006 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.621934    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljtgl\" (UniqueName: \"kubernetes.io/projected/a016483e-936f-45f5-a6c8-62bc13386af1-kube-api-access-ljtgl\") pod \"maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.753583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.753547    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:43.957547 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.957456    4555 generic.go:356] "Generic (PLEG): container finished" podID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerID="dbfd5a86527312d29c1c7629063838d63459c807e350689f4c3ab58cf952d7e4" exitCode=7
Oct 30 23:57:43.957547 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.957532    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x" event={"ID":"f7be6f05-8295-48e2-a176-a6ba4d22b085","Type":"ContainerDied","Data":"dbfd5a86527312d29c1c7629063838d63459c807e350689f4c3ab58cf952d7e4"}
Oct 30 23:57:43.957906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:43.957888    4555 scope.go:122] "RemoveContainer" containerID="dbfd5a86527312d29c1c7629063838d63459c807e350689f4c3ab58cf952d7e4"
Oct 30 23:57:44.068050 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:57:44.068020    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda016483e_936f_45f5_a6c8_62bc13386af1.slice/crio-4a7e59d9956a5ca64910ae8a25be37d092826b7b224021e2819dac4bace5de50 WatchSource:0}: Error finding container 4a7e59d9956a5ca64910ae8a25be37d092826b7b224021e2819dac4bace5de50: Status 404 returned error can't find the container with id 4a7e59d9956a5ca64910ae8a25be37d092826b7b224021e2819dac4bace5de50
Oct 30 23:57:44.069067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:44.069047    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"]
Oct 30 23:57:44.200956 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:44.200929    4555 scope.go:122] "RemoveContainer" containerID="6e66fe8d38cdc44ea570e55cb36e8f646a597306d12ffe8b45e4a868d3b330da"
Oct 30 23:57:44.980499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:44.980455    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" event={"ID":"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec","Type":"ContainerStarted","Data":"6f4cbac70614ea7e1473da60190c505783a403a886c237aa7da3859987207952"}
Oct 30 23:57:44.984690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:44.984658    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x" event={"ID":"f7be6f05-8295-48e2-a176-a6ba4d22b085","Type":"ContainerStarted","Data":"26262250979e2c70982bd6c5cec64afa3b66ea23f695efb68d4f0ed19c1cd945"}
Oct 30 23:57:44.987698 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:44.987672    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7" event={"ID":"a016483e-936f-45f5-a6c8-62bc13386af1","Type":"ContainerStarted","Data":"8400a2ec920336c1b490581dbffc18325978eb06343dd4320e28b0f47086dcba"}
Oct 30 23:57:44.987808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:44.987723    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7" event={"ID":"a016483e-936f-45f5-a6c8-62bc13386af1","Type":"ContainerStarted","Data":"4a7e59d9956a5ca64910ae8a25be37d092826b7b224021e2819dac4bace5de50"}
Oct 30 23:57:45.050478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:45.049104    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7" podStartSLOduration=2.049085779 podStartE2EDuration="2.049085779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:45.047136143 +0000 UTC m=+2323.973773739" watchObservedRunningTime="2026-10-30 23:57:45.049085779 +0000 UTC m=+2323.975723361"
Oct 30 23:57:45.124590 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:45.124513    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x" podStartSLOduration=4.124493156 podStartE2EDuration="4.124493156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:45.11263195 +0000 UTC m=+2324.039269532" watchObservedRunningTime="2026-10-30 23:57:45.124493156 +0000 UTC m=+2324.051130738"
Oct 30 23:57:46.003143 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.003050    4555 generic.go:356] "Generic (PLEG): container finished" podID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerID="26262250979e2c70982bd6c5cec64afa3b66ea23f695efb68d4f0ed19c1cd945" exitCode=7
Oct 30 23:57:46.003566 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.003175    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x" event={"ID":"f7be6f05-8295-48e2-a176-a6ba4d22b085","Type":"ContainerDied","Data":"26262250979e2c70982bd6c5cec64afa3b66ea23f695efb68d4f0ed19c1cd945"}
Oct 30 23:57:46.003566 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.003233    4555 scope.go:122] "RemoveContainer" containerID="dbfd5a86527312d29c1c7629063838d63459c807e350689f4c3ab58cf952d7e4"
Oct 30 23:57:46.003700 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.003638    4555 scope.go:122] "RemoveContainer" containerID="26262250979e2c70982bd6c5cec64afa3b66ea23f695efb68d4f0ed19c1cd945"
Oct 30 23:57:46.004201 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:46.004157    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-2x69x_odh-ai-gateway-infra(f7be6f05-8295-48e2-a176-a6ba4d22b085)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085"
Oct 30 23:57:46.005677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.005649    4555 generic.go:356] "Generic (PLEG): container finished" podID="a016483e-936f-45f5-a6c8-62bc13386af1" containerID="8400a2ec920336c1b490581dbffc18325978eb06343dd4320e28b0f47086dcba" exitCode=0
Oct 30 23:57:46.005808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.005678    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7" event={"ID":"a016483e-936f-45f5-a6c8-62bc13386af1","Type":"ContainerDied","Data":"8400a2ec920336c1b490581dbffc18325978eb06343dd4320e28b0f47086dcba"}
Oct 30 23:57:46.008622 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.008466    4555 generic.go:356] "Generic (PLEG): container finished" podID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerID="6f4cbac70614ea7e1473da60190c505783a403a886c237aa7da3859987207952" exitCode=7
Oct 30 23:57:46.008622 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.008524    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" event={"ID":"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec","Type":"ContainerDied","Data":"6f4cbac70614ea7e1473da60190c505783a403a886c237aa7da3859987207952"}
Oct 30 23:57:46.009162 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.009054    4555 scope.go:122] "RemoveContainer" containerID="6f4cbac70614ea7e1473da60190c505783a403a886c237aa7da3859987207952"
Oct 30 23:57:46.009638 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:46.009608    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f993bcbd-477cz_odh-ai-gateway-infra(9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec"
Oct 30 23:57:46.019136 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.019115    4555 scope.go:122] "RemoveContainer" containerID="6e66fe8d38cdc44ea570e55cb36e8f646a597306d12ffe8b45e4a868d3b330da"
Oct 30 23:57:46.029613 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:46.029591    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"]
Oct 30 23:57:47.025666 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.025634    4555 scope.go:122] "RemoveContainer" containerID="26262250979e2c70982bd6c5cec64afa3b66ea23f695efb68d4f0ed19c1cd945"
Oct 30 23:57:47.026102 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:47.025969    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-2x69x_odh-ai-gateway-infra(f7be6f05-8295-48e2-a176-a6ba4d22b085)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085"
Oct 30 23:57:47.320957 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.320922    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:47.323969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.323947    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:47.412447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.412413    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-openshift-ca\") pod \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") "
Oct 30 23:57:47.412631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.412527    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-kube-api-access-6xg52\" (UniqueName: \"kubernetes.io/projected/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-kube-api-access-6xg52\") pod \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") "
Oct 30 23:57:47.412631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.412587    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-openshift-ca\") pod \"a016483e-936f-45f5-a6c8-62bc13386af1\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") "
Oct 30 23:57:47.412722 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.412712    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-xks-ca\") pod \"a016483e-936f-45f5-a6c8-62bc13386af1\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") "
Oct 30 23:57:47.412770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.412757    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-xks-ca\") pod \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\" (UID: \"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec\") "
Oct 30 23:57:47.412811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.412798    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a016483e-936f-45f5-a6c8-62bc13386af1-kube-api-access-ljtgl\" (UniqueName: \"kubernetes.io/projected/a016483e-936f-45f5-a6c8-62bc13386af1-kube-api-access-ljtgl\") pod \"a016483e-936f-45f5-a6c8-62bc13386af1\" (UID: \"a016483e-936f-45f5-a6c8-62bc13386af1\") "
Oct 30 23:57:47.412876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.412847    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-openshift-ca" pod "9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" (UID: "9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:47.412923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.412906    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-openshift-ca" pod "a016483e-936f-45f5-a6c8-62bc13386af1" (UID: "a016483e-936f-45f5-a6c8-62bc13386af1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:47.413892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.413863    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:47.414037 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.413895    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:47.415704 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.415674    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-xks-ca" pod "9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" (UID: "9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:47.415838 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.415741    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a016483e-936f-45f5-a6c8-62bc13386af1-kube-api-access-ljtgl" pod "a016483e-936f-45f5-a6c8-62bc13386af1" (UID: "a016483e-936f-45f5-a6c8-62bc13386af1"). InnerVolumeSpecName "kube-api-access-ljtgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:47.419742 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.419717    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-xks-ca" pod "a016483e-936f-45f5-a6c8-62bc13386af1" (UID: "a016483e-936f-45f5-a6c8-62bc13386af1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:47.419884 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.419860    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-kube-api-access-6xg52" pod "9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" (UID: "9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec"). InnerVolumeSpecName "kube-api-access-6xg52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:47.515926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.515881    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xg52\" (UniqueName: \"kubernetes.io/projected/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-kube-api-access-6xg52\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:47.515926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.515922    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a016483e-936f-45f5-a6c8-62bc13386af1-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:47.515926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.515933    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:47.515926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:47.515943    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljtgl\" (UniqueName: \"kubernetes.io/projected/a016483e-936f-45f5-a6c8-62bc13386af1-kube-api-access-ljtgl\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:48.036794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.036764    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"
Oct 30 23:57:48.037277 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.036804    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz" event={"ID":"9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec","Type":"ContainerDied","Data":"ff34fa623d864428dfd5b7c4f2410d3c222e80eabeb2fa798b036c5c078872a1"}
Oct 30 23:57:48.037277 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.036877    4555 scope.go:122] "RemoveContainer" containerID="6f4cbac70614ea7e1473da60190c505783a403a886c237aa7da3859987207952"
Oct 30 23:57:48.038681 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.038595    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"
Oct 30 23:57:48.038681 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.038600    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7" event={"ID":"a016483e-936f-45f5-a6c8-62bc13386af1","Type":"ContainerDied","Data":"4a7e59d9956a5ca64910ae8a25be37d092826b7b224021e2819dac4bace5de50"}
Oct 30 23:57:48.038681 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.038690    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a7e59d9956a5ca64910ae8a25be37d092826b7b224021e2819dac4bace5de50"
Oct 30 23:57:48.080989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.080948    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"]
Oct 30 23:57:48.098089 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.092021    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-477cz"]
Oct 30 23:57:48.661655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.659415    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"]
Oct 30 23:57:48.661655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.660688    4555 cpu_manager.go:414] "containerMap: removing container" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerName="revoke-keys"
Oct 30 23:57:48.661655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.660714    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerName="revoke-keys"
Oct 30 23:57:48.661655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.660779    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a016483e-936f-45f5-a6c8-62bc13386af1" containerName="revoke-keys"
Oct 30 23:57:48.661655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.660789    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a016483e-936f-45f5-a6c8-62bc13386af1" containerName="revoke-keys"
Oct 30 23:57:48.661655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.660822    4555 cpu_manager.go:414] "containerMap: removing container" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerName="revoke-keys"
Oct 30 23:57:48.661655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.660831    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerName="revoke-keys"
Oct 30 23:57:48.661655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.661021    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerName="revoke-keys"
Oct 30 23:57:48.661655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.661034    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a016483e-936f-45f5-a6c8-62bc13386af1" containerName="revoke-keys"
Oct 30 23:57:48.661655 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.661053    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerName="revoke-keys"
Oct 30 23:57:48.670517 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.670490    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"]
Oct 30 23:57:48.670829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.670775    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:48.749921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.749875    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-6x9kk\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:48.750165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.749962    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmbn\" (UniqueName: \"kubernetes.io/projected/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-kube-api-access-msmbn\") pod \"maas-api-revoke-sub-f993bcbd-6x9kk\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:48.750165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.750013    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-6x9kk\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:48.858448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.854086    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-6x9kk\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:48.858448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.854170    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmbn\" (UniqueName: \"kubernetes.io/projected/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-kube-api-access-msmbn\") pod \"maas-api-revoke-sub-f993bcbd-6x9kk\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:48.858448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.854214    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-6x9kk\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:48.858448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.857832    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-6x9kk\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:48.858448 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.858169    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-6x9kk\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:48.953260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:48.953175    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmbn\" (UniqueName: \"kubernetes.io/projected/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-kube-api-access-msmbn\") pod \"maas-api-revoke-sub-f993bcbd-6x9kk\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:49.027980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.027931    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:57:49.167411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.167348    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85db5d76c9-xddv6"]
Oct 30 23:57:49.168900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.168848    4555 cpu_manager.go:414] "containerMap: removing container" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerName="revoke-keys"
Oct 30 23:57:49.168900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.168885    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerName="revoke-keys"
Oct 30 23:57:49.169224 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.169204    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" containerName="revoke-keys"
Oct 30 23:57:49.175278 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.175259    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.200558 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.197217    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85db5d76c9-xddv6"]
Oct 30 23:57:49.236513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.236413    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec" path="/var/lib/kubelet/pods/9e8ae5a9-9c20-4b55-a8cb-bde2e21057ec/volumes"
Oct 30 23:57:49.262389 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.262337    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-tmp\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.262581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.262486    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-trusted-ca-bundle\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.262581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.262513    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-service-ca\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.262581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.262548    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-oauth-serving-cert\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.262749 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.262618    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6c95\" (UniqueName: \"kubernetes.io/projected/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-kube-api-access-w6c95\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.262749 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.262638    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-console-serving-cert\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.262749 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.262693    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-console-config\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.262749 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.262733    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-console-oauth-config\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.363776 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.363739    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-console-config\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.363964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.363808    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-console-oauth-config\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.364066 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.364024    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-tmp\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.364156 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.364138    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-trusted-ca-bundle\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.364223 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.364176    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-service-ca\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.364223 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.364217    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-oauth-serving-cert\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.364380 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.364332    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6c95\" (UniqueName: \"kubernetes.io/projected/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-kube-api-access-w6c95\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.364458 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.364362    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-console-serving-cert\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.364586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.364558    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-console-config\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.365432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.365360    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-oauth-serving-cert\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.366186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.366148    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-service-ca\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.367230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.367052    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-trusted-ca-bundle\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.367396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.367351    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-console-oauth-config\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.367740 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.367715    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-tmp\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.372726 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.372701    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-console-serving-cert\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.374011 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:57:49.373985    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e4c4bf_0aa0_47c1_af55_ff8b73028512.slice/crio-4f6a4656a9df188866c85780c30cf7de3b5c1c0d3c52342fdf0747ff5bc1298e WatchSource:0}: Error finding container 4f6a4656a9df188866c85780c30cf7de3b5c1c0d3c52342fdf0747ff5bc1298e: Status 404 returned error can't find the container with id 4f6a4656a9df188866c85780c30cf7de3b5c1c0d3c52342fdf0747ff5bc1298e
Oct 30 23:57:49.375049 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.375029    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"]
Oct 30 23:57:49.384513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.384488    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6c95\" (UniqueName: \"kubernetes.io/projected/83ca2764-5d6e-4fbe-9af9-eb80bdf946b3-kube-api-access-w6c95\") pod \"console-85db5d76c9-xddv6\" (UID: \"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3\") " pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.506123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.506033    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:49.934632 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:49.934593    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85db5d76c9-xddv6"]
Oct 30 23:57:50.069505 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:50.069468    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85db5d76c9-xddv6" event={"ID":"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3","Type":"ContainerStarted","Data":"938e5437304b7af4e004725308097c9202010aaa5c39a90cebcccbc85c2735e1"}
Oct 30 23:57:50.071436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:50.071398    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" event={"ID":"f9e4c4bf-0aa0-47c1-af55-ff8b73028512","Type":"ContainerStarted","Data":"d5fdf3e64ad7c5c5d5c2435ca946b0e533d5f0787d26e8348782361a96c7a9b0"}
Oct 30 23:57:50.071630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:50.071613    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" event={"ID":"f9e4c4bf-0aa0-47c1-af55-ff8b73028512","Type":"ContainerStarted","Data":"4f6a4656a9df188866c85780c30cf7de3b5c1c0d3c52342fdf0747ff5bc1298e"}
Oct 30 23:57:50.094703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:50.094656    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" podStartSLOduration=2.094637949 podStartE2EDuration="2.094637949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:50.09081058 +0000 UTC m=+2329.017448152" watchObservedRunningTime="2026-10-30 23:57:50.094637949 +0000 UTC m=+2329.021275528"
Oct 30 23:57:51.086441 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.086362    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85db5d76c9-xddv6" event={"ID":"83ca2764-5d6e-4fbe-9af9-eb80bdf946b3","Type":"ContainerStarted","Data":"f6ecb9affca8eaf12aa1530a11b0ac9b934eefbd28058fcc83e31d7f74ecfc8c"}
Oct 30 23:57:51.088523 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.088493    4555 generic.go:356] "Generic (PLEG): container finished" podID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerID="d5fdf3e64ad7c5c5d5c2435ca946b0e533d5f0787d26e8348782361a96c7a9b0" exitCode=7
Oct 30 23:57:51.088669 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.088571    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" event={"ID":"f9e4c4bf-0aa0-47c1-af55-ff8b73028512","Type":"ContainerDied","Data":"d5fdf3e64ad7c5c5d5c2435ca946b0e533d5f0787d26e8348782361a96c7a9b0"}
Oct 30 23:57:51.088918 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.088903    4555 scope.go:122] "RemoveContainer" containerID="d5fdf3e64ad7c5c5d5c2435ca946b0e533d5f0787d26e8348782361a96c7a9b0"
Oct 30 23:57:51.113822 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.113774    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85db5d76c9-xddv6" podStartSLOduration=2.113757459 podStartE2EDuration="2.113757459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:51.107763917 +0000 UTC m=+2330.034401500" watchObservedRunningTime="2026-10-30 23:57:51.113757459 +0000 UTC m=+2330.040395067"
Oct 30 23:57:51.339744 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.339651    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"]
Oct 30 23:57:51.339927 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.339867    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy" containerID="cri-o://e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f" gracePeriod=30
Oct 30 23:57:51.345933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.345785    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"]
Oct 30 23:57:51.346095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.346033    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy" containerID="cri-o://5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b" gracePeriod=30
Oct 30 23:57:51.395071 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.395020    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"]
Oct 30 23:57:51.395300 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.395269    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerName="istio-proxy" containerID="cri-o://73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336" gracePeriod=30
Oct 30 23:57:51.422435 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.422306    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"]
Oct 30 23:57:51.422617 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.422567    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" containerID="cri-o://74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd" gracePeriod=30
Oct 30 23:57:51.429897 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.429864    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"]
Oct 30 23:57:51.430139 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.430112    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" containerID="cri-o://53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93" gracePeriod=30
Oct 30 23:57:51.437525 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.437497    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"]
Oct 30 23:57:51.437773 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.437740    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" containerID="cri-o://a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d" gracePeriod=30
Oct 30 23:57:51.445108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.445013    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"]
Oct 30 23:57:51.445304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.445256    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy" containerID="cri-o://0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84" gracePeriod=30
Oct 30 23:57:51.705299 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.705193    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"]
Oct 30 23:57:51.705514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.705472    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" containerID="cri-o://4cbe4578003fe9a98b72fca36b51f02ef1a331594fb38781a44ca0e9ab8a7c0c" gracePeriod=30
Oct 30 23:57:51.715548 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.715517    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"]
Oct 30 23:57:51.715744 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.715724    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy" containerID="cri-o://6c119b261e59d8273f86b0f31f75845ebda9eb5b2a684a7eba63cf968dd43ed2" gracePeriod=30
Oct 30 23:57:51.725975 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.725940    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"]
Oct 30 23:57:51.726237 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.726211    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy" containerID="cri-o://623aac7033a22e651ef10b7e7d067c2f121f04f60813a5ee1dcd35b87ad2447b" gracePeriod=30
Oct 30 23:57:51.734167 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.734131    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"]
Oct 30 23:57:51.734471 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:51.734434    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" containerID="cri-o://8637634dcced5385de37619eb15716d48de80ff0b0059ed80d03ac1673e10f7b" gracePeriod=30
Oct 30 23:57:52.104533 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:52.104482    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" event={"ID":"f9e4c4bf-0aa0-47c1-af55-ff8b73028512","Type":"ContainerStarted","Data":"085309b6f0c5a6254d0f387a092f499e8f62aebb38d89de58983906bd30781a7"}
Oct 30 23:57:52.261820 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:52.261781    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flxph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:57:52.261996 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:52.261840    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:57:53.004652 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:53.004609    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wnl2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:57:53.004869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:53.004676    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:57:53.125458 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:53.125418    4555 generic.go:356] "Generic (PLEG): container finished" podID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerID="085309b6f0c5a6254d0f387a092f499e8f62aebb38d89de58983906bd30781a7" exitCode=7
Oct 30 23:57:53.125930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:53.125529    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" event={"ID":"f9e4c4bf-0aa0-47c1-af55-ff8b73028512","Type":"ContainerDied","Data":"085309b6f0c5a6254d0f387a092f499e8f62aebb38d89de58983906bd30781a7"}
Oct 30 23:57:53.125930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:53.125635    4555 scope.go:122] "RemoveContainer" containerID="d5fdf3e64ad7c5c5d5c2435ca946b0e533d5f0787d26e8348782361a96c7a9b0"
Oct 30 23:57:53.125930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:53.125892    4555 scope.go:122] "RemoveContainer" containerID="085309b6f0c5a6254d0f387a092f499e8f62aebb38d89de58983906bd30781a7"
Oct 30 23:57:53.126471 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:53.126437    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f993bcbd-6x9kk_odh-ai-gateway-infra(f9e4c4bf-0aa0-47c1-af55-ff8b73028512)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512"
Oct 30 23:57:54.235436 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.231482    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"]
Oct 30 23:57:54.304483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.304145    4555 patch_prober.go:29] interesting pod/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:57:54.304483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.304213    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:57:54.326427 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.326315    4555 scope.go:122] "RemoveContainer" containerID="085309b6f0c5a6254d0f387a092f499e8f62aebb38d89de58983906bd30781a7"
Oct 30 23:57:54.348397 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:54.327157    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f993bcbd-6x9kk_odh-ai-gateway-infra(f9e4c4bf-0aa0-47c1-af55-ff8b73028512)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512"
Oct 30 23:57:54.415462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.415421    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-df3b5f-d42hm"]
Oct 30 23:57:54.496643 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.496538    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 23:57:54.663396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.655755    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.152:8000/ready\": context deadline exceeded"
Oct 30 23:57:54.663396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.660122    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:57:54.663891 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.663856    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:57:54.665397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.665347    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Oct 30 23:57:54.665648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:54.665628    4555 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:57:55.213691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:55.213655    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789133d1-763e-4f3f-aec7-72a5810a642b" path="/var/lib/kubelet/pods/789133d1-763e-4f3f-aec7-72a5810a642b/volumes"
Oct 30 23:57:56.109393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.106421    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"]
Oct 30 23:57:56.251394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.235634    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ce4ee8-8cs74"]
Oct 30 23:57:56.504601 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.504570    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:57:56.504823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.504643    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:57:56.543407 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.538317    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"]
Oct 30 23:57:56.546209 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.546186    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:56.548641 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:56.548601    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70538c08_5ce2_41fa_ac84_d6427beb113b.slice/crio-conmon-e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb9dbde_ccfc_4d1b_852d_1e7203497c07.slice/crio-conmon-5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc735d7_ab52_40b3_bece_087e83a5ee61.slice/crio-74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 23:57:56.548802 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:56.548774    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb9dbde_ccfc_4d1b_852d_1e7203497c07.slice/crio-5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc735d7_ab52_40b3_bece_087e83a5ee61.slice/crio-74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 23:57:56.552341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.552308    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"]
Oct 30 23:57:56.708559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.707798    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zbt\" (UniqueName: \"kubernetes.io/projected/3cc137c6-08f7-4080-a0df-c22951462c12-kube-api-access-c5zbt\") pod \"maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:56.708559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.708040    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:56.708559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.708122    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:56.811075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.809837    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zbt\" (UniqueName: \"kubernetes.io/projected/3cc137c6-08f7-4080-a0df-c22951462c12-kube-api-access-c5zbt\") pod \"maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:56.811075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.810076    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:56.811075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.810694    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:56.812515 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.811731    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:56.828587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.827515    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:56.858195 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.850127    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zbt\" (UniqueName: \"kubernetes.io/projected/3cc137c6-08f7-4080-a0df-c22951462c12-kube-api-access-c5zbt\") pod \"maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:56.858631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.858586    4555 patch_prober.go:29] interesting pod/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 23:57:56.858751 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.858646    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 23:57:56.997181 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:56.990030    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:57:57.149425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.138770    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:57:57.306815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.285075    4555 scope.go:122] "RemoveContainer" containerID="26262250979e2c70982bd6c5cec64afa3b66ea23f695efb68d4f0ed19c1cd945"
Oct 30 23:57:57.332738 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.332284    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:57:57.349019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.348972    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774a17c8-1193-431c-9319-8e30de37ead9" path="/var/lib/kubelet/pods/774a17c8-1193-431c-9319-8e30de37ead9/volumes"
Oct 30 23:57:57.354674 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.354532    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:57:57.371079 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.370500    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:57:57.397811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.397719    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.423690    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-certs\") pod \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.423748    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-data\") pod \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.423781    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70538c08-5ce2-41fa-ac84-d6427beb113b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70538c08-5ce2-41fa-ac84-d6427beb113b-istiod-ca-cert\") pod \"70538c08-5ce2-41fa-ac84-d6427beb113b\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.423813    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-token\" (UniqueName: \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-token\") pod \"c69a343e-ca93-47a1-892b-c17cb341cff0\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.423859    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-podinfo\") pod \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.423899    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-certs\") pod \"70538c08-5ce2-41fa-ac84-d6427beb113b\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.423937    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istiod-ca-cert\") pod \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.423963    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-data\") pod \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.423999    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-kube-api-access-vh4zc\" (UniqueName: \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-kube-api-access-vh4zc\") pod \"7fc735d7-ab52-40b3-bece-087e83a5ee61\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424029    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-envoy\") pod \"70538c08-5ce2-41fa-ac84-d6427beb113b\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424055    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istiod-ca-cert\") pod \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424139    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-token\" (UniqueName: \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-token\") pod \"70538c08-5ce2-41fa-ac84-d6427beb113b\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424170    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-envoy\") pod \"7fc735d7-ab52-40b3-bece-087e83a5ee61\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424205    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-podinfo\") pod \"70538c08-5ce2-41fa-ac84-d6427beb113b\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424235    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-token\" (UniqueName: \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-token\") pod \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424269    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-certs\") pod \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424345    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-data\") pod \"7fc735d7-ab52-40b3-bece-087e83a5ee61\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424397    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-socket\") pod \"c69a343e-ca93-47a1-892b-c17cb341cff0\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424483    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-token\" (UniqueName: \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-token\") pod \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424523    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-kube-api-access-hcs89\" (UniqueName: \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-kube-api-access-hcs89\") pod \"c69a343e-ca93-47a1-892b-c17cb341cff0\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424570    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-credential-socket\") pod \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424607    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-kube-api-access-snjx2\" (UniqueName: \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-kube-api-access-snjx2\") pod \"70538c08-5ce2-41fa-ac84-d6427beb113b\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424641    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-envoy\") pod \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424732    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-podinfo\") pod \"c69a343e-ca93-47a1-892b-c17cb341cff0\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424762    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-envoy\") pod \"c69a343e-ca93-47a1-892b-c17cb341cff0\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424815    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-envoy\") pod \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424874    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-credential-socket\") pod \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424913    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-certs\") pod \"7fc735d7-ab52-40b3-bece-087e83a5ee61\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.424971    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-credential-socket\") pod \"70538c08-5ce2-41fa-ac84-d6427beb113b\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425036    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c69a343e-ca93-47a1-892b-c17cb341cff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c69a343e-ca93-47a1-892b-c17cb341cff0-istiod-ca-cert\") pod \"c69a343e-ca93-47a1-892b-c17cb341cff0\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425074    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-socket\") pod \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425116    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-token\" (UniqueName: \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-token\") pod \"7fc735d7-ab52-40b3-bece-087e83a5ee61\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425156    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-kube-api-access-8gtc5\" (UniqueName: \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-kube-api-access-8gtc5\") pod \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425198    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-socket\") pod \"70538c08-5ce2-41fa-ac84-d6427beb113b\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425298    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-credential-socket\") pod \"7fc735d7-ab52-40b3-bece-087e83a5ee61\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425326    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-podinfo\") pod \"7fc735d7-ab52-40b3-bece-087e83a5ee61\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425415    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-data\") pod \"70538c08-5ce2-41fa-ac84-d6427beb113b\" (UID: \"70538c08-5ce2-41fa-ac84-d6427beb113b\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425493    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-data\") pod \"c69a343e-ca93-47a1-892b-c17cb341cff0\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425536    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-podinfo\") pod \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425588    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-socket\") pod \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\" (UID: \"4bb9dbde-ccfc-4d1b-852d-1e7203497c07\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425621    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-certs\") pod \"c69a343e-ca93-47a1-892b-c17cb341cff0\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425665    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-socket\") pod \"7fc735d7-ab52-40b3-bece-087e83a5ee61\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425705    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-kube-api-access-2j4b8\" (UniqueName: \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-kube-api-access-2j4b8\") pod \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\" (UID: \"473d73e9-7ff3-4f29-b8c6-5d40e71224af\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425739    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc735d7-ab52-40b3-bece-087e83a5ee61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc735d7-ab52-40b3-bece-087e83a5ee61-istiod-ca-cert\") pod \"7fc735d7-ab52-40b3-bece-087e83a5ee61\" (UID: \"7fc735d7-ab52-40b3-bece-087e83a5ee61\") "
Oct 30 23:57:57.426797 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.425787    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-credential-socket\") pod \"c69a343e-ca93-47a1-892b-c17cb341cff0\" (UID: \"c69a343e-ca93-47a1-892b-c17cb341cff0\") "
Oct 30 23:57:57.429761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.429167    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istiod-ca-cert" pod "473d73e9-7ff3-4f29-b8c6-5d40e71224af" (UID: "473d73e9-7ff3-4f29-b8c6-5d40e71224af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.429761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.429509    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-certs" pod "4bb9dbde-ccfc-4d1b-852d-1e7203497c07" (UID: "4bb9dbde-ccfc-4d1b-852d-1e7203497c07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.429925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.429844    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-socket" pod "4bb9dbde-ccfc-4d1b-852d-1e7203497c07" (UID: "4bb9dbde-ccfc-4d1b-852d-1e7203497c07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.430240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.430037    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-certs" pod "c69a343e-ca93-47a1-892b-c17cb341cff0" (UID: "c69a343e-ca93-47a1-892b-c17cb341cff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.430240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.430184    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.430240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.430209    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.430240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.430224    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.439350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.430258    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-socket" pod "7fc735d7-ab52-40b3-bece-087e83a5ee61" (UID: "7fc735d7-ab52-40b3-bece-087e83a5ee61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.439350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.430477    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-data" pod "473d73e9-7ff3-4f29-b8c6-5d40e71224af" (UID: "473d73e9-7ff3-4f29-b8c6-5d40e71224af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.439350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.430579    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-data" pod "70538c08-5ce2-41fa-ac84-d6427beb113b" (UID: "70538c08-5ce2-41fa-ac84-d6427beb113b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.439350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.430816    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70538c08-5ce2-41fa-ac84-d6427beb113b-istiod-ca-cert" pod "70538c08-5ce2-41fa-ac84-d6427beb113b" (UID: "70538c08-5ce2-41fa-ac84-d6427beb113b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.439350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.431461    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc735d7-ab52-40b3-bece-087e83a5ee61-istiod-ca-cert" pod "7fc735d7-ab52-40b3-bece-087e83a5ee61" (UID: "7fc735d7-ab52-40b3-bece-087e83a5ee61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.439884 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.439821    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-socket" pod "70538c08-5ce2-41fa-ac84-d6427beb113b" (UID: "70538c08-5ce2-41fa-ac84-d6427beb113b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.448258 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.447953    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-socket" pod "c69a343e-ca93-47a1-892b-c17cb341cff0" (UID: "c69a343e-ca93-47a1-892b-c17cb341cff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.450474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.449427    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-certs" pod "70538c08-5ce2-41fa-ac84-d6427beb113b" (UID: "70538c08-5ce2-41fa-ac84-d6427beb113b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.450474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.449808    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istiod-ca-cert" pod "4bb9dbde-ccfc-4d1b-852d-1e7203497c07" (UID: "4bb9dbde-ccfc-4d1b-852d-1e7203497c07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.450474 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.450092    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-data" pod "4bb9dbde-ccfc-4d1b-852d-1e7203497c07" (UID: "4bb9dbde-ccfc-4d1b-852d-1e7203497c07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.456847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.456703    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-credential-socket" pod "c69a343e-ca93-47a1-892b-c17cb341cff0" (UID: "c69a343e-ca93-47a1-892b-c17cb341cff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.461675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.457238    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-data" pod "c69a343e-ca93-47a1-892b-c17cb341cff0" (UID: "c69a343e-ca93-47a1-892b-c17cb341cff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.461675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.457282    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-credential-socket" pod "7fc735d7-ab52-40b3-bece-087e83a5ee61" (UID: "7fc735d7-ab52-40b3-bece-087e83a5ee61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.461675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.457312    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-data" pod "7fc735d7-ab52-40b3-bece-087e83a5ee61" (UID: "7fc735d7-ab52-40b3-bece-087e83a5ee61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.461675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.457861    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-kube-api-access-snjx2" pod "70538c08-5ce2-41fa-ac84-d6427beb113b" (UID: "70538c08-5ce2-41fa-ac84-d6427beb113b"). InnerVolumeSpecName "kube-api-access-snjx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.461675 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.458948    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-certs" pod "473d73e9-7ff3-4f29-b8c6-5d40e71224af" (UID: "473d73e9-7ff3-4f29-b8c6-5d40e71224af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.462350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.462329    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-credential-socket" pod "473d73e9-7ff3-4f29-b8c6-5d40e71224af" (UID: "473d73e9-7ff3-4f29-b8c6-5d40e71224af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.463934 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.463732    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-credential-socket" pod "4bb9dbde-ccfc-4d1b-852d-1e7203497c07" (UID: "4bb9dbde-ccfc-4d1b-852d-1e7203497c07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.464392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.464342    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69a343e-ca93-47a1-892b-c17cb341cff0-istiod-ca-cert" pod "c69a343e-ca93-47a1-892b-c17cb341cff0" (UID: "c69a343e-ca93-47a1-892b-c17cb341cff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.464887 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.464867    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-socket" pod "473d73e9-7ff3-4f29-b8c6-5d40e71224af" (UID: "473d73e9-7ff3-4f29-b8c6-5d40e71224af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.470706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.470508    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-certs" pod "7fc735d7-ab52-40b3-bece-087e83a5ee61" (UID: "7fc735d7-ab52-40b3-bece-087e83a5ee61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.477264 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.477213    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-credential-socket" pod "70538c08-5ce2-41fa-ac84-d6427beb113b" (UID: "70538c08-5ce2-41fa-ac84-d6427beb113b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.532573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.532533    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc735d7-ab52-40b3-bece-087e83a5ee61-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.532756 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.532743    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.532842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.532834    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.532906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.532899    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70538c08-5ce2-41fa-ac84-d6427beb113b-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.532987    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533061    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533129 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533118    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533188 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533180    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533249 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533240    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533318    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533408    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533484    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snjx2\" (UniqueName: \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-kube-api-access-snjx2\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533561 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533554    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533617    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533685 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533678    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533736 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533728    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c69a343e-ca93-47a1-892b-c17cb341cff0-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533790    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533854 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533845    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533902    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.533973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.533965    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.534057 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.534048    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.534119 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.534111    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.534171 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.534165    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.554444 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:57.554136    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:57:57.586154 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.585032    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:57:57.587561 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.586958    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-kube-api-access-vh4zc" pod "7fc735d7-ab52-40b3-bece-087e83a5ee61" (UID: "7fc735d7-ab52-40b3-bece-087e83a5ee61"). InnerVolumeSpecName "kube-api-access-vh4zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.587561 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.587058    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-token" pod "7fc735d7-ab52-40b3-bece-087e83a5ee61" (UID: "7fc735d7-ab52-40b3-bece-087e83a5ee61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.598819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.598250    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-token" pod "4bb9dbde-ccfc-4d1b-852d-1e7203497c07" (UID: "4bb9dbde-ccfc-4d1b-852d-1e7203497c07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.598819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.598567    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-podinfo" pod "473d73e9-7ff3-4f29-b8c6-5d40e71224af" (UID: "473d73e9-7ff3-4f29-b8c6-5d40e71224af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.599014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.598864    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-token" pod "473d73e9-7ff3-4f29-b8c6-5d40e71224af" (UID: "473d73e9-7ff3-4f29-b8c6-5d40e71224af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.608581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.608485    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-kube-api-access-2j4b8" pod "473d73e9-7ff3-4f29-b8c6-5d40e71224af" (UID: "473d73e9-7ff3-4f29-b8c6-5d40e71224af"). InnerVolumeSpecName "kube-api-access-2j4b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.610442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.610357    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-envoy" pod "70538c08-5ce2-41fa-ac84-d6427beb113b" (UID: "70538c08-5ce2-41fa-ac84-d6427beb113b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.610809 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.610767    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-kube-api-access-hcs89" pod "c69a343e-ca93-47a1-892b-c17cb341cff0" (UID: "c69a343e-ca93-47a1-892b-c17cb341cff0"). InnerVolumeSpecName "kube-api-access-hcs89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.611039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.611005    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-envoy" pod "c69a343e-ca93-47a1-892b-c17cb341cff0" (UID: "c69a343e-ca93-47a1-892b-c17cb341cff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.611304 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.611272    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-kube-api-access-8gtc5" pod "4bb9dbde-ccfc-4d1b-852d-1e7203497c07" (UID: "4bb9dbde-ccfc-4d1b-852d-1e7203497c07"). InnerVolumeSpecName "kube-api-access-8gtc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.613276 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.613251    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-podinfo" pod "7fc735d7-ab52-40b3-bece-087e83a5ee61" (UID: "7fc735d7-ab52-40b3-bece-087e83a5ee61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.613566 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.613547    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-envoy" pod "7fc735d7-ab52-40b3-bece-087e83a5ee61" (UID: "7fc735d7-ab52-40b3-bece-087e83a5ee61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.613671 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.613623    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-token" pod "c69a343e-ca93-47a1-892b-c17cb341cff0" (UID: "c69a343e-ca93-47a1-892b-c17cb341cff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.613753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.613735    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-podinfo" pod "70538c08-5ce2-41fa-ac84-d6427beb113b" (UID: "70538c08-5ce2-41fa-ac84-d6427beb113b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.613828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.613798    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-envoy" pod "473d73e9-7ff3-4f29-b8c6-5d40e71224af" (UID: "473d73e9-7ff3-4f29-b8c6-5d40e71224af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.613888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.613863    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-token" pod "70538c08-5ce2-41fa-ac84-d6427beb113b" (UID: "70538c08-5ce2-41fa-ac84-d6427beb113b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.613956 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.613683    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-podinfo" pod "c69a343e-ca93-47a1-892b-c17cb341cff0" (UID: "c69a343e-ca93-47a1-892b-c17cb341cff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.614455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.614432    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-podinfo" pod "4bb9dbde-ccfc-4d1b-852d-1e7203497c07" (UID: "4bb9dbde-ccfc-4d1b-852d-1e7203497c07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.623519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.623077    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-envoy" pod "4bb9dbde-ccfc-4d1b-852d-1e7203497c07" (UID: "4bb9dbde-ccfc-4d1b-852d-1e7203497c07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.631240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.630872    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:57:57.638113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.634632    4555 generic.go:356] "Generic (PLEG): container finished" podID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerID="4cbe4578003fe9a98b72fca36b51f02ef1a331594fb38781a44ca0e9ab8a7c0c" exitCode=0
Oct 30 23:57:57.638113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.634711    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" event={"ID":"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5","Type":"ContainerDied","Data":"4cbe4578003fe9a98b72fca36b51f02ef1a331594fb38781a44ca0e9ab8a7c0c"}
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640220    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640252    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640270    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640284    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640297    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640312    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gtc5\" (UniqueName: \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-kube-api-access-8gtc5\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640326    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640339    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640352    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j4b8\" (UniqueName: \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-kube-api-access-2j4b8\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640391    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640407    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640421    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh4zc\" (UniqueName: \"kubernetes.io/projected/7fc735d7-ab52-40b3-bece-087e83a5ee61-kube-api-access-vh4zc\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640434    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640446    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640461    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc735d7-ab52-40b3-bece-087e83a5ee61-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.640467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640474    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70538c08-5ce2-41fa-ac84-d6427beb113b-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.641146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640490    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb9dbde-ccfc-4d1b-852d-1e7203497c07-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.641146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640504    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473d73e9-7ff3-4f29-b8c6-5d40e71224af-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.641146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.640517    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcs89\" (UniqueName: \"kubernetes.io/projected/c69a343e-ca93-47a1-892b-c17cb341cff0-kube-api-access-hcs89\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.642595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.642024    4555 generic.go:356] "Generic (PLEG): container finished" podID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerID="623aac7033a22e651ef10b7e7d067c2f121f04f60813a5ee1dcd35b87ad2447b" exitCode=0
Oct 30 23:57:57.642595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.642128    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" event={"ID":"b8aee7e5-9755-4dee-8000-31115262a4c3","Type":"ContainerDied","Data":"623aac7033a22e651ef10b7e7d067c2f121f04f60813a5ee1dcd35b87ad2447b"}
Oct 30 23:57:57.643958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.643891    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:57:57.646467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.644176    4555 generic.go:356] "Generic (PLEG): container finished" podID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerID="e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f" exitCode=0
Oct 30 23:57:57.646467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.644252    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" event={"ID":"70538c08-5ce2-41fa-ac84-d6427beb113b","Type":"ContainerDied","Data":"e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f"}
Oct 30 23:57:57.646467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.644284    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb" event={"ID":"70538c08-5ce2-41fa-ac84-d6427beb113b","Type":"ContainerDied","Data":"3d0dbb8499d9b0818d4aaab2ad174e3e11f121a37c6cc2747eec7780051058dc"}
Oct 30 23:57:57.646467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.644306    4555 scope.go:122] "RemoveContainer" containerID="e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f"
Oct 30 23:57:57.646467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.644427    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"
Oct 30 23:57:57.650240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.648826    4555 generic.go:356] "Generic (PLEG): container finished" podID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerID="74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd" exitCode=0
Oct 30 23:57:57.650240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.648962    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" event={"ID":"7fc735d7-ab52-40b3-bece-087e83a5ee61","Type":"ContainerDied","Data":"74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd"}
Oct 30 23:57:57.650240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.649018    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph" event={"ID":"7fc735d7-ab52-40b3-bece-087e83a5ee61","Type":"ContainerDied","Data":"c1fb97d5664c634aff4fadec9a812c7d9c7c35f381d81a1d47713777fc108ad3"}
Oct 30 23:57:57.650240 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.649111    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"
Oct 30 23:57:57.689819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.681655    4555 generic.go:356] "Generic (PLEG): container finished" podID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerID="73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336" exitCode=0
Oct 30 23:57:57.689819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.681773    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" event={"ID":"c69a343e-ca93-47a1-892b-c17cb341cff0","Type":"ContainerDied","Data":"73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336"}
Oct 30 23:57:57.689819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.681824    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx" event={"ID":"c69a343e-ca93-47a1-892b-c17cb341cff0","Type":"ContainerDied","Data":"3ed2a60ef76ca7e3f918ab7c779d00e192ddeb25ad217c84798ed2a79225eabc"}
Oct 30 23:57:57.689819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.684668    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"
Oct 30 23:57:57.712806 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.712777    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:57:57.716360 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.716337    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:57:57.717194 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.717164    4555 generic.go:356] "Generic (PLEG): container finished" podID="16b24d54-b21b-4168-a520-24717ced83a7" containerID="53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93" exitCode=0
Oct 30 23:57:57.717315 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.717299    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" event={"ID":"16b24d54-b21b-4168-a520-24717ced83a7","Type":"ContainerDied","Data":"53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93"}
Oct 30 23:57:57.717485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.717350    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n" event={"ID":"16b24d54-b21b-4168-a520-24717ced83a7","Type":"ContainerDied","Data":"ef4b83a67e12a2c87b0549045f9168c0550cda53dfb66ab78a547f441b735b2f"}
Oct 30 23:57:57.717574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.717503    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"
Oct 30 23:57:57.721203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.721177    4555 generic.go:356] "Generic (PLEG): container finished" podID="ca2043db-3020-4355-b7d0-49f148914780" containerID="6c119b261e59d8273f86b0f31f75845ebda9eb5b2a684a7eba63cf968dd43ed2" exitCode=0
Oct 30 23:57:57.721334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.721316    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx" event={"ID":"ca2043db-3020-4355-b7d0-49f148914780","Type":"ContainerDied","Data":"6c119b261e59d8273f86b0f31f75845ebda9eb5b2a684a7eba63cf968dd43ed2"}
Oct 30 23:57:57.721548 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.721296    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"
Oct 30 23:57:57.725396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.723259    4555 generic.go:356] "Generic (PLEG): container finished" podID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerID="5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b" exitCode=0
Oct 30 23:57:57.725396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.723347    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" event={"ID":"4bb9dbde-ccfc-4d1b-852d-1e7203497c07","Type":"ContainerDied","Data":"5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b"}
Oct 30 23:57:57.725396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.723404    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj" event={"ID":"4bb9dbde-ccfc-4d1b-852d-1e7203497c07","Type":"ContainerDied","Data":"ceb602b36c763560da5280fb948f90d7dd6371c3421c879f0976e64ffbc493d8"}
Oct 30 23:57:57.725396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.723467    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"
Oct 30 23:57:57.735094 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.735069    4555 scope.go:122] "RemoveContainer" containerID="e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f"
Oct 30 23:57:57.740021 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:57.739876    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f\": container with ID starting with e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f not found: ID does not exist" containerID="e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f"
Oct 30 23:57:57.740021 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.739918    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f"} err="failed to get container status \"e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f\": rpc error: code = NotFound desc = could not find container \"e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f\": container with ID starting with e1944fa7ae41fc5d50e71e51def4f0f1e78eb18ce41e5d45e06723e92777482f not found: ID does not exist"
Oct 30 23:57:57.740021 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.739945    4555 scope.go:122] "RemoveContainer" containerID="74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd"
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.741700    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-token\" (UniqueName: \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-token\") pod \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.741785    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-kube-api-access-mcqd9\" (UniqueName: \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-kube-api-access-mcqd9\") pod \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.741822    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istiod-ca-cert\") pod \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.741853    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-credential-socket\") pod \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.741927    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-token\" (UniqueName: \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-token\") pod \"165550ee-3ccc-485a-bbe3-d7e7d295253d\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.741957    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca2043db-3020-4355-b7d0-49f148914780-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2043db-3020-4355-b7d0-49f148914780-istio-podinfo\") pod \"ca2043db-3020-4355-b7d0-49f148914780\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.741994    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-socket\") pod \"165550ee-3ccc-485a-bbe3-d7e7d295253d\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.742023    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b24d54-b21b-4168-a520-24717ced83a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b24d54-b21b-4168-a520-24717ced83a7-istiod-ca-cert\") pod \"16b24d54-b21b-4168-a520-24717ced83a7\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.742058    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2043db-3020-4355-b7d0-49f148914780-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2043db-3020-4355-b7d0-49f148914780-istiod-ca-cert\") pod \"ca2043db-3020-4355-b7d0-49f148914780\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.742090    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-socket\") pod \"16b24d54-b21b-4168-a520-24717ced83a7\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.742154    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-podinfo\") pod \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.742194    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-data\") pod \"16b24d54-b21b-4168-a520-24717ced83a7\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.742234    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-certs\") pod \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.742288    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-podinfo\") pod \"165550ee-3ccc-485a-bbe3-d7e7d295253d\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") "
Oct 30 23:57:57.743480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.742337    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-kube-api-access-dj5lb\" (UniqueName: \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-kube-api-access-dj5lb\") pod \"ca2043db-3020-4355-b7d0-49f148914780\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.744576    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-data\") pod \"ca2043db-3020-4355-b7d0-49f148914780\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.744652    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-kube-api-access-ws2pp\" (UniqueName: \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-kube-api-access-ws2pp\") pod \"16b24d54-b21b-4168-a520-24717ced83a7\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.744695    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-socket\") pod \"ca2043db-3020-4355-b7d0-49f148914780\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.744739    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-istio-token\" (UniqueName: \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-istio-token\") pod \"ca2043db-3020-4355-b7d0-49f148914780\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.744784    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-socket\") pod \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.744820    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-envoy\") pod \"16b24d54-b21b-4168-a520-24717ced83a7\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.744928    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-data\") pod \"165550ee-3ccc-485a-bbe3-d7e7d295253d\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.744992    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-certs\") pod \"165550ee-3ccc-485a-bbe3-d7e7d295253d\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.745042    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-credential-socket\") pod \"ca2043db-3020-4355-b7d0-49f148914780\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.745088    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-envoy\") pod \"165550ee-3ccc-485a-bbe3-d7e7d295253d\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.745127    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-credential-socket\") pod \"16b24d54-b21b-4168-a520-24717ced83a7\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.745165    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-kube-api-access-j7fwv\" (UniqueName: \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-kube-api-access-j7fwv\") pod \"165550ee-3ccc-485a-bbe3-d7e7d295253d\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") "
Oct 30 23:57:57.745306 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.745243    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-istio-token\" (UniqueName: \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-istio-token\") pod \"16b24d54-b21b-4168-a520-24717ced83a7\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") "
Oct 30 23:57:57.749244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.745277    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-envoy\") pod \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") "
Oct 30 23:57:57.749244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.748466    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16b24d54-b21b-4168-a520-24717ced83a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b24d54-b21b-4168-a520-24717ced83a7-istio-podinfo\") pod \"16b24d54-b21b-4168-a520-24717ced83a7\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") "
Oct 30 23:57:57.749244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.748522    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-envoy\") pod \"ca2043db-3020-4355-b7d0-49f148914780\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") "
Oct 30 23:57:57.749244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.748599    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-certs\") pod \"ca2043db-3020-4355-b7d0-49f148914780\" (UID: \"ca2043db-3020-4355-b7d0-49f148914780\") "
Oct 30 23:57:57.749244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.748634    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-certs\") pod \"16b24d54-b21b-4168-a520-24717ced83a7\" (UID: \"16b24d54-b21b-4168-a520-24717ced83a7\") "
Oct 30 23:57:57.749244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.748695    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-data\") pod \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\" (UID: \"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5\") "
Oct 30 23:57:57.749244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.748733    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165550ee-3ccc-485a-bbe3-d7e7d295253d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165550ee-3ccc-485a-bbe3-d7e7d295253d-istiod-ca-cert\") pod \"165550ee-3ccc-485a-bbe3-d7e7d295253d\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") "
Oct 30 23:57:57.749244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.748765    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-credential-socket\") pod \"165550ee-3ccc-485a-bbe3-d7e7d295253d\" (UID: \"165550ee-3ccc-485a-bbe3-d7e7d295253d\") "
Oct 30 23:57:57.749244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.749062    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-socket" pod "ca2043db-3020-4355-b7d0-49f148914780" (UID: "ca2043db-3020-4355-b7d0-49f148914780"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.750022 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.749834    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-credential-socket" pod "165550ee-3ccc-485a-bbe3-d7e7d295253d" (UID: "165550ee-3ccc-485a-bbe3-d7e7d295253d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.750116 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.750024    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-data" pod "ca2043db-3020-4355-b7d0-49f148914780" (UID: "ca2043db-3020-4355-b7d0-49f148914780"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.750392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.750343    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.750643 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.750490    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.750723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.750684    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istiod-ca-cert" pod "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" (UID: "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.750874 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.750821    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-data" pod "165550ee-3ccc-485a-bbe3-d7e7d295253d" (UID: "165550ee-3ccc-485a-bbe3-d7e7d295253d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.750944 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.750914    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-certs" pod "165550ee-3ccc-485a-bbe3-d7e7d295253d" (UID: "165550ee-3ccc-485a-bbe3-d7e7d295253d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.751017 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.750938    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-credential-socket" pod "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" (UID: "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.751437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.751270    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-credential-socket" pod "ca2043db-3020-4355-b7d0-49f148914780" (UID: "ca2043db-3020-4355-b7d0-49f148914780"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.751902 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.751853    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-certs" pod "16b24d54-b21b-4168-a520-24717ced83a7" (UID: "16b24d54-b21b-4168-a520-24717ced83a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.752725 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.752064    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-data" pod "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" (UID: "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.752725 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.752689    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-certs" pod "ca2043db-3020-4355-b7d0-49f148914780" (UID: "ca2043db-3020-4355-b7d0-49f148914780"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.752874 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.752800    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165550ee-3ccc-485a-bbe3-d7e7d295253d-istiod-ca-cert" pod "165550ee-3ccc-485a-bbe3-d7e7d295253d" (UID: "165550ee-3ccc-485a-bbe3-d7e7d295253d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.753171 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.753145    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-token" pod "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" (UID: "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.753769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.753599    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2043db-3020-4355-b7d0-49f148914780-istiod-ca-cert" pod "ca2043db-3020-4355-b7d0-49f148914780" (UID: "ca2043db-3020-4355-b7d0-49f148914780"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.754639 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.754335    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-credential-socket" pod "16b24d54-b21b-4168-a520-24717ced83a7" (UID: "16b24d54-b21b-4168-a520-24717ced83a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.755803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.755292    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-socket" pod "165550ee-3ccc-485a-bbe3-d7e7d295253d" (UID: "165550ee-3ccc-485a-bbe3-d7e7d295253d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.755803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.755538    4555 generic.go:356] "Generic (PLEG): container finished" podID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerID="8637634dcced5385de37619eb15716d48de80ff0b0059ed80d03ac1673e10f7b" exitCode=0
Oct 30 23:57:57.755803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.755644    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" event={"ID":"e49e8515-359e-483c-81aa-51c63c2e37e3","Type":"ContainerDied","Data":"8637634dcced5385de37619eb15716d48de80ff0b0059ed80d03ac1673e10f7b"}
Oct 30 23:57:57.756002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.755906    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b24d54-b21b-4168-a520-24717ced83a7-istiod-ca-cert" pod "16b24d54-b21b-4168-a520-24717ced83a7" (UID: "16b24d54-b21b-4168-a520-24717ced83a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.756657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.756275    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-socket" pod "16b24d54-b21b-4168-a520-24717ced83a7" (UID: "16b24d54-b21b-4168-a520-24717ced83a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.756657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.756498    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-data" pod "16b24d54-b21b-4168-a520-24717ced83a7" (UID: "16b24d54-b21b-4168-a520-24717ced83a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.758097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.756992    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-socket" pod "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" (UID: "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.758097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.757482    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-envoy" pod "ca2043db-3020-4355-b7d0-49f148914780" (UID: "ca2043db-3020-4355-b7d0-49f148914780"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.759778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.757985    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-certs" pod "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" (UID: "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.759778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.758473    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-kube-api-access-j7fwv" pod "165550ee-3ccc-485a-bbe3-d7e7d295253d" (UID: "165550ee-3ccc-485a-bbe3-d7e7d295253d"). InnerVolumeSpecName "kube-api-access-j7fwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.759778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.758540    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca2043db-3020-4355-b7d0-49f148914780-istio-podinfo" pod "ca2043db-3020-4355-b7d0-49f148914780" (UID: "ca2043db-3020-4355-b7d0-49f148914780"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.760780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.760745    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-kube-api-access-ws2pp" pod "16b24d54-b21b-4168-a520-24717ced83a7" (UID: "16b24d54-b21b-4168-a520-24717ced83a7"). InnerVolumeSpecName "kube-api-access-ws2pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.764336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.763072    4555 generic.go:356] "Generic (PLEG): container finished" podID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerID="a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d" exitCode=0
Oct 30 23:57:57.764336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.763172    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" event={"ID":"473d73e9-7ff3-4f29-b8c6-5d40e71224af","Type":"ContainerDied","Data":"a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d"}
Oct 30 23:57:57.764336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.763225    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q" event={"ID":"473d73e9-7ff3-4f29-b8c6-5d40e71224af","Type":"ContainerDied","Data":"8f4aa5f20bb1be05673322881c2766ae46c5f9f5b81d51c647aeb95926e912dc"}
Oct 30 23:57:57.764336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.763314    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"
Oct 30 23:57:57.772511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.772431    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-envoy" pod "165550ee-3ccc-485a-bbe3-d7e7d295253d" (UID: "165550ee-3ccc-485a-bbe3-d7e7d295253d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.773761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.773150    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-istio-token" pod "ca2043db-3020-4355-b7d0-49f148914780" (UID: "ca2043db-3020-4355-b7d0-49f148914780"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.773761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.773602    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-podinfo" pod "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" (UID: "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.773761 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.773725    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-envoy" pod "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" (UID: "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.775037 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.774998    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-kube-api-access-dj5lb" pod "ca2043db-3020-4355-b7d0-49f148914780" (UID: "ca2043db-3020-4355-b7d0-49f148914780"). InnerVolumeSpecName "kube-api-access-dj5lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.775609 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.775575    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16b24d54-b21b-4168-a520-24717ced83a7-istio-podinfo" pod "16b24d54-b21b-4168-a520-24717ced83a7" (UID: "16b24d54-b21b-4168-a520-24717ced83a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.780532 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.776432    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-istio-token" pod "16b24d54-b21b-4168-a520-24717ced83a7" (UID: "16b24d54-b21b-4168-a520-24717ced83a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.780532 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.777641    4555 scope.go:122] "RemoveContainer" containerID="74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd"
Oct 30 23:57:57.780532 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.778515    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-token" pod "165550ee-3ccc-485a-bbe3-d7e7d295253d" (UID: "165550ee-3ccc-485a-bbe3-d7e7d295253d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.780532 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.778606    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-kube-api-access-mcqd9" pod "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" (UID: "8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5"). InnerVolumeSpecName "kube-api-access-mcqd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.780947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.780732    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-podinfo" pod "165550ee-3ccc-485a-bbe3-d7e7d295253d" (UID: "165550ee-3ccc-485a-bbe3-d7e7d295253d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.783629 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.783591    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-envoy" pod "16b24d54-b21b-4168-a520-24717ced83a7" (UID: "16b24d54-b21b-4168-a520-24717ced83a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.786006 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:57.785961    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd\": container with ID starting with 74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd not found: ID does not exist" containerID="74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd"
Oct 30 23:57:57.786192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.786122    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd"} err="failed to get container status \"74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd\": rpc error: code = NotFound desc = could not find container \"74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd\": container with ID starting with 74a62861d3ec466596e8b6df36fb1fa6dafa24bf8df9e3f0652459a7639fd2fd not found: ID does not exist"
Oct 30 23:57:57.786192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.786152    4555 scope.go:122] "RemoveContainer" containerID="73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336"
Oct 30 23:57:57.792175 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.791803    4555 generic.go:356] "Generic (PLEG): container finished" podID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerID="0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84" exitCode=0
Oct 30 23:57:57.792175 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.792010    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"
Oct 30 23:57:57.792175 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.792114    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" event={"ID":"165550ee-3ccc-485a-bbe3-d7e7d295253d","Type":"ContainerDied","Data":"0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84"}
Oct 30 23:57:57.792175 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.792162    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w" event={"ID":"165550ee-3ccc-485a-bbe3-d7e7d295253d","Type":"ContainerDied","Data":"1419de60580e2a1012608e2cdbea13b41b6112a2d587b06172a4ea3e791d72c3"}
Oct 30 23:57:57.813481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.812735    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:57:57.818132 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.818059    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:57:57.822023 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.822007    4555 scope.go:122] "RemoveContainer" containerID="73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336"
Oct 30 23:57:57.822479 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:57.822460    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336\": container with ID starting with 73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336 not found: ID does not exist" containerID="73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336"
Oct 30 23:57:57.822550 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.822488    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336"} err="failed to get container status \"73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336\": rpc error: code = NotFound desc = could not find container \"73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336\": container with ID starting with 73fceafe92a85dc675b723b8e85e1d8596cc50d590fcc6707d25fffb7d5ee336 not found: ID does not exist"
Oct 30 23:57:57.822550 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.822505    4555 scope.go:122] "RemoveContainer" containerID="53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93"
Oct 30 23:57:57.825624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.825604    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"]
Oct 30 23:57:57.833581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.833548    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-cwtmx"]
Oct 30 23:57:57.836618 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.836599    4555 scope.go:122] "RemoveContainer" containerID="53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93"
Oct 30 23:57:57.836970 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:57.836954    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93\": container with ID starting with 53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93 not found: ID does not exist" containerID="53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93"
Oct 30 23:57:57.837092 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.837075    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93"} err="failed to get container status \"53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93\": rpc error: code = NotFound desc = could not find container \"53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93\": container with ID starting with 53dddc39282750218ed8ba783bc1bd3fbe1cb6c38ae33579d3cb5aa7b29c3b93 not found: ID does not exist"
Oct 30 23:57:57.837165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.837157    4555 scope.go:122] "RemoveContainer" containerID="6c119b261e59d8273f86b0f31f75845ebda9eb5b2a684a7eba63cf968dd43ed2"
Oct 30 23:57:57.852963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.852415    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-kube-api-access-nhdqc\" (UniqueName: \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-kube-api-access-nhdqc\") pod \"b8aee7e5-9755-4dee-8000-31115262a4c3\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") "
Oct 30 23:57:57.852963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.852459    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-data\") pod \"b8aee7e5-9755-4dee-8000-31115262a4c3\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") "
Oct 30 23:57:57.852963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.852571    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-token\" (UniqueName: \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-token\") pod \"b8aee7e5-9755-4dee-8000-31115262a4c3\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") "
Oct 30 23:57:57.852963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.852629    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49e8515-359e-483c-81aa-51c63c2e37e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49e8515-359e-483c-81aa-51c63c2e37e3-istiod-ca-cert\") pod \"e49e8515-359e-483c-81aa-51c63c2e37e3\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") "
Oct 30 23:57:57.852963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.852660    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-credential-socket\") pod \"e49e8515-359e-483c-81aa-51c63c2e37e3\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") "
Oct 30 23:57:57.852963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.852715    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-token\" (UniqueName: \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-token\") pod \"e49e8515-359e-483c-81aa-51c63c2e37e3\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") "
Oct 30 23:57:57.852963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.852763    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-podinfo\") pod \"b8aee7e5-9755-4dee-8000-31115262a4c3\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") "
Oct 30 23:57:57.852963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.852806    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-credential-socket\") pod \"b8aee7e5-9755-4dee-8000-31115262a4c3\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") "
Oct 30 23:57:57.861959 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.861908    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-envoy\") pod \"b8aee7e5-9755-4dee-8000-31115262a4c3\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") "
Oct 30 23:57:57.862095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.862049    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-certs\") pod \"b8aee7e5-9755-4dee-8000-31115262a4c3\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") "
Oct 30 23:57:57.862095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.862077    4555 scope.go:122] "RemoveContainer" containerID="5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b"
Oct 30 23:57:57.862238 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.862119    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-kube-api-access-6h46l\" (UniqueName: \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-kube-api-access-6h46l\") pod \"e49e8515-359e-483c-81aa-51c63c2e37e3\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") "
Oct 30 23:57:57.862301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.862229    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-podinfo\") pod \"e49e8515-359e-483c-81aa-51c63c2e37e3\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") "
Oct 30 23:57:57.862301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.862295    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-socket\") pod \"b8aee7e5-9755-4dee-8000-31115262a4c3\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") "
Oct 30 23:57:57.862454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.862403    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-certs\") pod \"e49e8515-359e-483c-81aa-51c63c2e37e3\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") "
Oct 30 23:57:57.862519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.862504    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-envoy\") pod \"e49e8515-359e-483c-81aa-51c63c2e37e3\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") "
Oct 30 23:57:57.862634 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.862615    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-socket\") pod \"e49e8515-359e-483c-81aa-51c63c2e37e3\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") "
Oct 30 23:57:57.862691 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.862664    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-data\") pod \"e49e8515-359e-483c-81aa-51c63c2e37e3\" (UID: \"e49e8515-359e-483c-81aa-51c63c2e37e3\") "
Oct 30 23:57:57.862740 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.862719    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8aee7e5-9755-4dee-8000-31115262a4c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8aee7e5-9755-4dee-8000-31115262a4c3-istiod-ca-cert\") pod \"b8aee7e5-9755-4dee-8000-31115262a4c3\" (UID: \"b8aee7e5-9755-4dee-8000-31115262a4c3\") "
Oct 30 23:57:57.864394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864351    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864400    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864416    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864441    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864455    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864471    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7fwv\" (UniqueName: \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-kube-api-access-j7fwv\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864487    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864506    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864520    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b24d54-b21b-4168-a520-24717ced83a7-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864535    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864549    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864569    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864850    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864883    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165550ee-3ccc-485a-bbe3-d7e7d295253d-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864899    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864921    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcqd9\" (UniqueName: \"kubernetes.io/projected/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-kube-api-access-mcqd9\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864935    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864952    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864971    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.864987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.864986    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2043db-3020-4355-b7d0-49f148914780-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865009    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165550ee-3ccc-485a-bbe3-d7e7d295253d-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865023    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b24d54-b21b-4168-a520-24717ced83a7-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865043    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2043db-3020-4355-b7d0-49f148914780-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865057    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865072    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865087    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865104    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865119    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165550ee-3ccc-485a-bbe3-d7e7d295253d-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865137    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj5lb\" (UniqueName: \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-kube-api-access-dj5lb\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865150    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2043db-3020-4355-b7d0-49f148914780-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865169    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws2pp\" (UniqueName: \"kubernetes.io/projected/16b24d54-b21b-4168-a520-24717ced83a7-kube-api-access-ws2pp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865184    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2043db-3020-4355-b7d0-49f148914780-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865199    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.865552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865215    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b24d54-b21b-4168-a520-24717ced83a7-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.877416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865625    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8aee7e5-9755-4dee-8000-31115262a4c3-istiod-ca-cert" pod "b8aee7e5-9755-4dee-8000-31115262a4c3" (UID: "b8aee7e5-9755-4dee-8000-31115262a4c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.877416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.865907    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-certs" pod "b8aee7e5-9755-4dee-8000-31115262a4c3" (UID: "b8aee7e5-9755-4dee-8000-31115262a4c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.877416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.876143    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49e8515-359e-483c-81aa-51c63c2e37e3-istiod-ca-cert" pod "e49e8515-359e-483c-81aa-51c63c2e37e3" (UID: "e49e8515-359e-483c-81aa-51c63c2e37e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:57.877416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.876931    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-credential-socket" pod "e49e8515-359e-483c-81aa-51c63c2e37e3" (UID: "e49e8515-359e-483c-81aa-51c63c2e37e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.880166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.879976    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-envoy" pod "b8aee7e5-9755-4dee-8000-31115262a4c3" (UID: "b8aee7e5-9755-4dee-8000-31115262a4c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.880513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.880482    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-credential-socket" pod "b8aee7e5-9755-4dee-8000-31115262a4c3" (UID: "b8aee7e5-9755-4dee-8000-31115262a4c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.888136 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.887751    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-data" pod "b8aee7e5-9755-4dee-8000-31115262a4c3" (UID: "b8aee7e5-9755-4dee-8000-31115262a4c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.888136 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.887890    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"]
Oct 30 23:57:57.888300 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.888261    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-socket" pod "e49e8515-359e-483c-81aa-51c63c2e37e3" (UID: "e49e8515-359e-483c-81aa-51c63c2e37e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.888913 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.888536    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-socket" pod "b8aee7e5-9755-4dee-8000-31115262a4c3" (UID: "b8aee7e5-9755-4dee-8000-31115262a4c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.888913 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.888704    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-certs" pod "e49e8515-359e-483c-81aa-51c63c2e37e3" (UID: "e49e8515-359e-483c-81aa-51c63c2e37e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.889331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.889305    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-data" pod "e49e8515-359e-483c-81aa-51c63c2e37e3" (UID: "e49e8515-359e-483c-81aa-51c63c2e37e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.895866 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.895840    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flxph"]
Oct 30 23:57:57.897067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.897042    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-envoy" pod "e49e8515-359e-483c-81aa-51c63c2e37e3" (UID: "e49e8515-359e-483c-81aa-51c63c2e37e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:57.900477 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.900449    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-kube-api-access-6h46l" pod "e49e8515-359e-483c-81aa-51c63c2e37e3" (UID: "e49e8515-359e-483c-81aa-51c63c2e37e3"). InnerVolumeSpecName "kube-api-access-6h46l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.900674 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.900651    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-kube-api-access-nhdqc" pod "b8aee7e5-9755-4dee-8000-31115262a4c3" (UID: "b8aee7e5-9755-4dee-8000-31115262a4c3"). InnerVolumeSpecName "kube-api-access-nhdqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.900796 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.900780    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-token" pod "e49e8515-359e-483c-81aa-51c63c2e37e3" (UID: "e49e8515-359e-483c-81aa-51c63c2e37e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.902928 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.900907    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-token" pod "b8aee7e5-9755-4dee-8000-31115262a4c3" (UID: "b8aee7e5-9755-4dee-8000-31115262a4c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:57.903933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.903893    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-podinfo" pod "b8aee7e5-9755-4dee-8000-31115262a4c3" (UID: "b8aee7e5-9755-4dee-8000-31115262a4c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.904925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.904078    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-podinfo" pod "e49e8515-359e-483c-81aa-51c63c2e37e3" (UID: "e49e8515-359e-483c-81aa-51c63c2e37e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:57.912723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.912695    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"]
Oct 30 23:57:57.918919 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.918892    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-2pjlb"]
Oct 30 23:57:57.922813 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.922788    4555 scope.go:122] "RemoveContainer" containerID="5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b"
Oct 30 23:57:57.923236 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:57.923212    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b\": container with ID starting with 5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b not found: ID does not exist" containerID="5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b"
Oct 30 23:57:57.923318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.923248    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b"} err="failed to get container status \"5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b\": rpc error: code = NotFound desc = could not find container \"5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b\": container with ID starting with 5bca9e6b45e415675ba6f0b7cbe39d1ee9c044a1ac8eb47a08fd8b9ab097310b not found: ID does not exist"
Oct 30 23:57:57.923318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.923272    4555 scope.go:122] "RemoveContainer" containerID="a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d"
Oct 30 23:57:57.925440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.925409    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"]
Oct 30 23:57:57.930885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.930864    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-4qrfj"]
Oct 30 23:57:57.935921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.935897    4555 scope.go:122] "RemoveContainer" containerID="a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d"
Oct 30 23:57:57.936341 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:57.936287    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d\": container with ID starting with a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d not found: ID does not exist" containerID="a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d"
Oct 30 23:57:57.936452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.936342    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d"} err="failed to get container status \"a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d\": rpc error: code = NotFound desc = could not find container \"a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d\": container with ID starting with a0e1289c29e2c79f834421d8adad10179bbcca9c06b81b77a6cd5fbe4348138d not found: ID does not exist"
Oct 30 23:57:57.936452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.936431    4555 scope.go:122] "RemoveContainer" containerID="0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84"
Oct 30 23:57:57.938635 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:57:57.938613    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc137c6_08f7_4080_a0df_c22951462c12.slice/crio-2347fea88288839e7baa6a415d28846d2559eb0747e6174fbccbeff3f2976118 WatchSource:0}: Error finding container 2347fea88288839e7baa6a415d28846d2559eb0747e6174fbccbeff3f2976118: Status 404 returned error can't find the container with id 2347fea88288839e7baa6a415d28846d2559eb0747e6174fbccbeff3f2976118
Oct 30 23:57:57.940739 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.940680    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"]
Oct 30 23:57:57.946555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.946526    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"]
Oct 30 23:57:57.966305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.962526    4555 scope.go:122] "RemoveContainer" containerID="0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84"
Oct 30 23:57:57.966305 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:57.963283    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84\": container with ID starting with 0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84 not found: ID does not exist" containerID="0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84"
Oct 30 23:57:57.966305 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.963325    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84"} err="failed to get container status \"0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84\": rpc error: code = NotFound desc = could not find container \"0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84\": container with ID starting with 0a265c0e05b5be3001db6e368e3a2ab93e7b68731a9fc3c9e972e61e387ece84 not found: ID does not exist"
Oct 30 23:57:57.968383 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.968341    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.968805 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.968785    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h46l\" (UniqueName: \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-kube-api-access-6h46l\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.968923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.968909    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969018 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969006    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969113 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969103    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969180    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969264 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969254    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969345 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969335    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969439    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8aee7e5-9755-4dee-8000-31115262a4c3-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969536 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969525    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdqc\" (UniqueName: \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-kube-api-access-nhdqc\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969656 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969644    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969741 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969730    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969822    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49e8515-359e-483c-81aa-51c63c2e37e3-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969899    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49e8515-359e-483c-81aa-51c63c2e37e3-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.969988 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.969979    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49e8515-359e-483c-81aa-51c63c2e37e3-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.970062 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.970051    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.970137 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.970127    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.970217 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.970207    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8aee7e5-9755-4dee-8000-31115262a4c3-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:57.971231 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:57.971210    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-tpx4q"]
Oct 30 23:57:58.008149 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.008118    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"]
Oct 30 23:57:58.014613 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.014582    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-8295w"]
Oct 30 23:57:58.052864 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.052840    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"]
Oct 30 23:57:58.060840 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.060811    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wnl2n"]
Oct 30 23:57:58.071446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.071420    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"]
Oct 30 23:57:58.079024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.078999    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-s97fx"]
Oct 30 23:57:58.806495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.806450    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv" event={"ID":"8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5","Type":"ContainerDied","Data":"78561450a27d68eb3281b77d7800089c00ee89b08fe408bf36c42938ab0dc9af"}
Oct 30 23:57:58.806978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.806495    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"
Oct 30 23:57:58.806978 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.806614    4555 scope.go:122] "RemoveContainer" containerID="4cbe4578003fe9a98b72fca36b51f02ef1a331594fb38781a44ca0e9ab8a7c0c"
Oct 30 23:57:58.811397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.808592    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"
Oct 30 23:57:58.811397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.810605    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq" event={"ID":"b8aee7e5-9755-4dee-8000-31115262a4c3","Type":"ContainerDied","Data":"72734eb11b4df5019c4e3de0c133b310d6e6831628fba6bef7228420bf78e4fb"}
Oct 30 23:57:58.812807 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.812774    4555 generic.go:356] "Generic (PLEG): container finished" podID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerID="10ee67428addbd240aa49d1234ebb39306b7291821986e07beff43794a7e06ce" exitCode=7
Oct 30 23:57:58.812912 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.812868    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x" event={"ID":"f7be6f05-8295-48e2-a176-a6ba4d22b085","Type":"ContainerDied","Data":"10ee67428addbd240aa49d1234ebb39306b7291821986e07beff43794a7e06ce"}
Oct 30 23:57:58.813460 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.813362    4555 scope.go:122] "RemoveContainer" containerID="10ee67428addbd240aa49d1234ebb39306b7291821986e07beff43794a7e06ce"
Oct 30 23:57:58.813886 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:57:58.813852    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-2x69x_odh-ai-gateway-infra(f7be6f05-8295-48e2-a176-a6ba4d22b085)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085"
Oct 30 23:57:58.824545 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.824487    4555 scope.go:122] "RemoveContainer" containerID="623aac7033a22e651ef10b7e7d067c2f121f04f60813a5ee1dcd35b87ad2447b"
Oct 30 23:57:58.824727 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.824704    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk" event={"ID":"e49e8515-359e-483c-81aa-51c63c2e37e3","Type":"ContainerDied","Data":"3c2b6388eb0285c58fb57af43e8741a1677f790f6cc01f96dc875aff9c5140b7"}
Oct 30 23:57:58.824805 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.824721    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"
Oct 30 23:57:58.832437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.832409    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx" event={"ID":"3cc137c6-08f7-4080-a0df-c22951462c12","Type":"ContainerStarted","Data":"b289419cf4f2aefb6b337120595c377195d1d8651e7b93a76a2947ebfaf50f0f"}
Oct 30 23:57:58.832535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.832459    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx" event={"ID":"3cc137c6-08f7-4080-a0df-c22951462c12","Type":"ContainerStarted","Data":"2347fea88288839e7baa6a415d28846d2559eb0747e6174fbccbeff3f2976118"}
Oct 30 23:57:58.840347 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.840325    4555 scope.go:122] "RemoveContainer" containerID="26262250979e2c70982bd6c5cec64afa3b66ea23f695efb68d4f0ed19c1cd945"
Oct 30 23:57:58.855623 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.855603    4555 scope.go:122] "RemoveContainer" containerID="8637634dcced5385de37619eb15716d48de80ff0b0059ed80d03ac1673e10f7b"
Oct 30 23:57:58.865025 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.864974    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx" podStartSLOduration=2.86496116 podStartE2EDuration="2.86496116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:58.860179356 +0000 UTC m=+2337.786816938" watchObservedRunningTime="2026-10-30 23:57:58.86496116 +0000 UTC m=+2337.791598741"
Oct 30 23:57:58.880746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.880714    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"]
Oct 30 23:57:58.885614 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.885591    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-q8zxk"]
Oct 30 23:57:58.895680 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.895656    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"]
Oct 30 23:57:58.912041 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.912005    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-75bvq"]
Oct 30 23:57:58.917126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.917098    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"]
Oct 30 23:57:58.921780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:58.921714    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-zpvpv"]
Oct 30 23:57:59.574108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.574069    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" path="/var/lib/kubelet/pods/165550ee-3ccc-485a-bbe3-d7e7d295253d/volumes"
Oct 30 23:57:59.575995 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.575972    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b24d54-b21b-4168-a520-24717ced83a7" path="/var/lib/kubelet/pods/16b24d54-b21b-4168-a520-24717ced83a7/volumes"
Oct 30 23:57:59.592818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.592775    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" path="/var/lib/kubelet/pods/473d73e9-7ff3-4f29-b8c6-5d40e71224af/volumes"
Oct 30 23:57:59.593822 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.593800    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" path="/var/lib/kubelet/pods/4bb9dbde-ccfc-4d1b-852d-1e7203497c07/volumes"
Oct 30 23:57:59.594909 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.594783    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" path="/var/lib/kubelet/pods/70538c08-5ce2-41fa-ac84-d6427beb113b/volumes"
Oct 30 23:57:59.596204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.596184    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" path="/var/lib/kubelet/pods/7fc735d7-ab52-40b3-bece-087e83a5ee61/volumes"
Oct 30 23:57:59.597030 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.596975    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" path="/var/lib/kubelet/pods/8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5/volumes"
Oct 30 23:57:59.597800 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.597782    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" path="/var/lib/kubelet/pods/b8aee7e5-9755-4dee-8000-31115262a4c3/volumes"
Oct 30 23:57:59.599024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.599006    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" path="/var/lib/kubelet/pods/c69a343e-ca93-47a1-892b-c17cb341cff0/volumes"
Oct 30 23:57:59.600786 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.600767    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2043db-3020-4355-b7d0-49f148914780" path="/var/lib/kubelet/pods/ca2043db-3020-4355-b7d0-49f148914780/volumes"
Oct 30 23:57:59.601667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.601650    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" path="/var/lib/kubelet/pods/e49e8515-359e-483c-81aa-51c63c2e37e3/volumes"
Oct 30 23:57:59.618010 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.617965    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:59.618227 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.618213    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:59.618507 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.618412    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:59.625583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.625560    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85db5d76c9-xddv6"
Oct 30 23:57:59.735175 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.735135    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7845b665bc-52x8q"]
Oct 30 23:57:59.895705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.895613    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"]
Oct 30 23:57:59.898152 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.897600    4555 generic.go:356] "Generic (PLEG): container finished" podID="3cc137c6-08f7-4080-a0df-c22951462c12" containerID="b289419cf4f2aefb6b337120595c377195d1d8651e7b93a76a2947ebfaf50f0f" exitCode=0
Oct 30 23:57:59.898152 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:57:59.897719    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx" event={"ID":"3cc137c6-08f7-4080-a0df-c22951462c12","Type":"ContainerDied","Data":"b289419cf4f2aefb6b337120595c377195d1d8651e7b93a76a2947ebfaf50f0f"}
Oct 30 23:58:00.169473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.169408    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:58:00.270591 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.270557    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-openshift-ca\") pod \"f7be6f05-8295-48e2-a176-a6ba4d22b085\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") "
Oct 30 23:58:00.270764 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.270669    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-xks-ca\") pod \"f7be6f05-8295-48e2-a176-a6ba4d22b085\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") "
Oct 30 23:58:00.270911 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.270892    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7be6f05-8295-48e2-a176-a6ba4d22b085-kube-api-access-vmrhf\" (UniqueName: \"kubernetes.io/projected/f7be6f05-8295-48e2-a176-a6ba4d22b085-kube-api-access-vmrhf\") pod \"f7be6f05-8295-48e2-a176-a6ba4d22b085\" (UID: \"f7be6f05-8295-48e2-a176-a6ba4d22b085\") "
Oct 30 23:58:00.271063 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.270964    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-openshift-ca" pod "f7be6f05-8295-48e2-a176-a6ba4d22b085" (UID: "f7be6f05-8295-48e2-a176-a6ba4d22b085"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:00.271923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.271894    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:00.273028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.272991    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7be6f05-8295-48e2-a176-a6ba4d22b085-kube-api-access-vmrhf" pod "f7be6f05-8295-48e2-a176-a6ba4d22b085" (UID: "f7be6f05-8295-48e2-a176-a6ba4d22b085"). InnerVolumeSpecName "kube-api-access-vmrhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:00.273127 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.273103    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-xks-ca" pod "f7be6f05-8295-48e2-a176-a6ba4d22b085" (UID: "f7be6f05-8295-48e2-a176-a6ba4d22b085"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:00.373808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.373775    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmrhf\" (UniqueName: \"kubernetes.io/projected/f7be6f05-8295-48e2-a176-a6ba4d22b085-kube-api-access-vmrhf\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:00.373808 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.373809    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be6f05-8295-48e2-a176-a6ba4d22b085-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:00.914838 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.914804    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"
Oct 30 23:58:00.914838 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.914824    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x" event={"ID":"f7be6f05-8295-48e2-a176-a6ba4d22b085","Type":"ContainerDied","Data":"2e079b6d546919773bbf5f8a2f4bbc3eff292bc68e12345f3f9ce1bf653f5584"}
Oct 30 23:58:00.915445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.914890    4555 scope.go:122] "RemoveContainer" containerID="10ee67428addbd240aa49d1234ebb39306b7291821986e07beff43794a7e06ce"
Oct 30 23:58:00.953765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.953514    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"]
Oct 30 23:58:00.960426 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:00.958871    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-2x69x"]
Oct 30 23:58:01.168475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.164991    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:58:01.196619 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.196582    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-xks-ca\") pod \"3cc137c6-08f7-4080-a0df-c22951462c12\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") "
Oct 30 23:58:01.196839 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.196716    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc137c6-08f7-4080-a0df-c22951462c12-kube-api-access-c5zbt\" (UniqueName: \"kubernetes.io/projected/3cc137c6-08f7-4080-a0df-c22951462c12-kube-api-access-c5zbt\") pod \"3cc137c6-08f7-4080-a0df-c22951462c12\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") "
Oct 30 23:58:01.196914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.196849    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-openshift-ca\") pod \"3cc137c6-08f7-4080-a0df-c22951462c12\" (UID: \"3cc137c6-08f7-4080-a0df-c22951462c12\") "
Oct 30 23:58:01.198020 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.197988    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-openshift-ca" pod "3cc137c6-08f7-4080-a0df-c22951462c12" (UID: "3cc137c6-08f7-4080-a0df-c22951462c12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:01.214726 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.214698    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc137c6-08f7-4080-a0df-c22951462c12-kube-api-access-c5zbt" pod "3cc137c6-08f7-4080-a0df-c22951462c12" (UID: "3cc137c6-08f7-4080-a0df-c22951462c12"). InnerVolumeSpecName "kube-api-access-c5zbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:01.214843 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.214757    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-xks-ca" pod "3cc137c6-08f7-4080-a0df-c22951462c12" (UID: "3cc137c6-08f7-4080-a0df-c22951462c12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:01.227475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.227443    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085" path="/var/lib/kubelet/pods/f7be6f05-8295-48e2-a176-a6ba4d22b085/volumes"
Oct 30 23:58:01.299492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.299464    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5zbt\" (UniqueName: \"kubernetes.io/projected/3cc137c6-08f7-4080-a0df-c22951462c12-kube-api-access-c5zbt\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:01.299492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.299489    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:01.299492 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.299500    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc137c6-08f7-4080-a0df-c22951462c12-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:01.930003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.929974    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"
Oct 30 23:58:01.930482 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.929973    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx" event={"ID":"3cc137c6-08f7-4080-a0df-c22951462c12","Type":"ContainerDied","Data":"2347fea88288839e7baa6a415d28846d2559eb0747e6174fbccbeff3f2976118"}
Oct 30 23:58:01.930482 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:01.930183    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2347fea88288839e7baa6a415d28846d2559eb0747e6174fbccbeff3f2976118"
Oct 30 23:58:02.035487 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.035445    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"]
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036516    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036538    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036557    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036565    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036589    4555 cpu_manager.go:414] "containerMap: removing container" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036597    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036607    4555 cpu_manager.go:414] "containerMap: removing container" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036616    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036634    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036642    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036665    4555 cpu_manager.go:414] "containerMap: removing container" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036672    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036684    4555 cpu_manager.go:414] "containerMap: removing container" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036691    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036708    4555 cpu_manager.go:414] "containerMap: removing container" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036715    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036728    4555 cpu_manager.go:414] "containerMap: removing container" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036735    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036751    4555 cpu_manager.go:414] "containerMap: removing container" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036758    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036773    4555 cpu_manager.go:414] "containerMap: removing container" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036780    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036793    4555 cpu_manager.go:414] "containerMap: removing container" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036800    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036812    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036818    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036855    4555 cpu_manager.go:414] "containerMap: removing container" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036862    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036877    4555 cpu_manager.go:414] "containerMap: removing container" podUID="3cc137c6-08f7-4080-a0df-c22951462c12" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.036884    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc137c6-08f7-4080-a0df-c22951462c12" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037042    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="473d73e9-7ff3-4f29-b8c6-5d40e71224af" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037056    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037067    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2043db-3020-4355-b7d0-49f148914780" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037076    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="165550ee-3ccc-485a-bbe3-d7e7d295253d" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037088    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69a343e-ca93-47a1-892b-c17cb341cff0" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037099    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="70538c08-5ce2-41fa-ac84-d6427beb113b" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037111    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49e8515-359e-483c-81aa-51c63c2e37e3" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037123    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9d3e62-ce88-426b-a0ba-9d41d1ddbaf5" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037134    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b24d54-b21b-4168-a520-24717ced83a7" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037144    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc735d7-ab52-40b3-bece-087e83a5ee61" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037159    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb9dbde-ccfc-4d1b-852d-1e7203497c07" containerName="istio-proxy"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037169    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc137c6-08f7-4080-a0df-c22951462c12" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037177    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037185    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7be6f05-8295-48e2-a176-a6ba4d22b085" containerName="revoke-keys"
Oct 30 23:58:02.039281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.037196    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8aee7e5-9755-4dee-8000-31115262a4c3" containerName="istio-proxy"
Oct 30 23:58:02.046417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.044128    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.051263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.051238    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"]
Oct 30 23:58:02.258817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.256206    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-cd8m4\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.258817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.256295    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6x7\" (UniqueName: \"kubernetes.io/projected/66366b53-e15d-4e4e-959b-f5b3a6adf586-kube-api-access-6l6x7\") pod \"maas-api-revoke-sub-91354e50-cd8m4\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.258817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.256492    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-cd8m4\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.445251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.442987    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-cd8m4\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.445251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.443205    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-cd8m4\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.445251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.443287    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6x7\" (UniqueName: \"kubernetes.io/projected/66366b53-e15d-4e4e-959b-f5b3a6adf586-kube-api-access-6l6x7\") pod \"maas-api-revoke-sub-91354e50-cd8m4\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.445251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.444799    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-cd8m4\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.461855 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.461816    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-cd8m4\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.465828 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.465798    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6x7\" (UniqueName: \"kubernetes.io/projected/66366b53-e15d-4e4e-959b-f5b3a6adf586-kube-api-access-6l6x7\") pod \"maas-api-revoke-sub-91354e50-cd8m4\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.517101 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.517025    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:02.807286 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:58:02.807255    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66366b53_e15d_4e4e_959b_f5b3a6adf586.slice/crio-282aa9f47b3d1873868203fdf814a26481de89dc922bb0c78faf497f52dfe967 WatchSource:0}: Error finding container 282aa9f47b3d1873868203fdf814a26481de89dc922bb0c78faf497f52dfe967: Status 404 returned error can't find the container with id 282aa9f47b3d1873868203fdf814a26481de89dc922bb0c78faf497f52dfe967
Oct 30 23:58:02.807779 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.807754    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"]
Oct 30 23:58:02.963234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:02.963146    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4" event={"ID":"66366b53-e15d-4e4e-959b-f5b3a6adf586","Type":"ContainerStarted","Data":"282aa9f47b3d1873868203fdf814a26481de89dc922bb0c78faf497f52dfe967"}
Oct 30 23:58:03.985930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:03.985889    4555 generic.go:356] "Generic (PLEG): container finished" podID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerID="d0bd19a30247231089af65d72dd047e4f51dcd7d963f72ad312cc5b1c58ea3b8" exitCode=7
Oct 30 23:58:03.986454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:03.986177    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4" event={"ID":"66366b53-e15d-4e4e-959b-f5b3a6adf586","Type":"ContainerDied","Data":"d0bd19a30247231089af65d72dd047e4f51dcd7d963f72ad312cc5b1c58ea3b8"}
Oct 30 23:58:03.986796 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:03.986770    4555 scope.go:122] "RemoveContainer" containerID="d0bd19a30247231089af65d72dd047e4f51dcd7d963f72ad312cc5b1c58ea3b8"
Oct 30 23:58:04.998196 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:04.998157    4555 generic.go:356] "Generic (PLEG): container finished" podID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerID="7f1ec5aaa7c1ec624571e01bc1b9d5dfafe63ec7b8d9f29754969ecba2d91ceb" exitCode=7
Oct 30 23:58:04.998724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:04.998255    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4" event={"ID":"66366b53-e15d-4e4e-959b-f5b3a6adf586","Type":"ContainerDied","Data":"7f1ec5aaa7c1ec624571e01bc1b9d5dfafe63ec7b8d9f29754969ecba2d91ceb"}
Oct 30 23:58:04.998724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:04.998326    4555 scope.go:122] "RemoveContainer" containerID="d0bd19a30247231089af65d72dd047e4f51dcd7d963f72ad312cc5b1c58ea3b8"
Oct 30 23:58:04.998827 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:04.998742    4555 scope.go:122] "RemoveContainer" containerID="7f1ec5aaa7c1ec624571e01bc1b9d5dfafe63ec7b8d9f29754969ecba2d91ceb"
Oct 30 23:58:04.999199 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:04.999176    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-cd8m4_odh-ai-gateway-infra(66366b53-e15d-4e4e-959b-f5b3a6adf586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586"
Oct 30 23:58:06.008859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:06.008827    4555 scope.go:122] "RemoveContainer" containerID="7f1ec5aaa7c1ec624571e01bc1b9d5dfafe63ec7b8d9f29754969ecba2d91ceb"
Oct 30 23:58:06.009426 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:06.009314    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-cd8m4_odh-ai-gateway-infra(66366b53-e15d-4e4e-959b-f5b3a6adf586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586"
Oct 30 23:58:06.200821 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:06.200792    4555 scope.go:122] "RemoveContainer" containerID="085309b6f0c5a6254d0f387a092f499e8f62aebb38d89de58983906bd30781a7"
Oct 30 23:58:07.070394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:07.067820    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" event={"ID":"f9e4c4bf-0aa0-47c1-af55-ff8b73028512","Type":"ContainerStarted","Data":"9c0381d2a95cc08a73db53907aa0c2e013f545d2bf2a91f4a5ac6b8ee3c154a5"}
Oct 30 23:58:08.080044 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:08.080003    4555 generic.go:356] "Generic (PLEG): container finished" podID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerID="9c0381d2a95cc08a73db53907aa0c2e013f545d2bf2a91f4a5ac6b8ee3c154a5" exitCode=7
Oct 30 23:58:08.080561 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:08.080044    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" event={"ID":"f9e4c4bf-0aa0-47c1-af55-ff8b73028512","Type":"ContainerDied","Data":"9c0381d2a95cc08a73db53907aa0c2e013f545d2bf2a91f4a5ac6b8ee3c154a5"}
Oct 30 23:58:08.080561 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:08.080118    4555 scope.go:122] "RemoveContainer" containerID="085309b6f0c5a6254d0f387a092f499e8f62aebb38d89de58983906bd30781a7"
Oct 30 23:58:08.080677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:08.080627    4555 scope.go:122] "RemoveContainer" containerID="9c0381d2a95cc08a73db53907aa0c2e013f545d2bf2a91f4a5ac6b8ee3c154a5"
Oct 30 23:58:08.081088 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:08.081062    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f993bcbd-6x9kk_odh-ai-gateway-infra(f9e4c4bf-0aa0-47c1-af55-ff8b73028512)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512"
Oct 30 23:58:08.122863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:08.122827    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"]
Oct 30 23:58:09.169769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.169729    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"]
Oct 30 23:58:09.204881 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.204850    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"]
Oct 30 23:58:09.207499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.205140    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.213697 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.213662    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e5934f-serving-cert\""
Oct 30 23:58:09.353445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.336563    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-metrics-certs\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.353445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.336751    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b9vj\" (UniqueName: \"kubernetes.io/projected/ca96124c-888e-4d50-b285-ef517527688a-kube-api-access-7b9vj\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.353445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.337016    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.353445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.337087    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca96124c-888e-4d50-b285-ef517527688a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.438719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.438677    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-metrics-certs\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.438918 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.438830    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b9vj\" (UniqueName: \"kubernetes.io/projected/ca96124c-888e-4d50-b285-ef517527688a-kube-api-access-7b9vj\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.439084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.439065    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.439161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.439128    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca96124c-888e-4d50-b285-ef517527688a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.440045 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.440021    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca96124c-888e-4d50-b285-ef517527688a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.443277 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.443255    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.443903 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.443882    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:58:09.444461 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.443935    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-metrics-certs\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.462406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.462359    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b9vj\" (UniqueName: \"kubernetes.io/projected/ca96124c-888e-4d50-b285-ef517527688a-kube-api-access-7b9vj\") pod \"maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.527150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.527111    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:09.540157 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.540127    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-kube-api-access-msmbn\" (UniqueName: \"kubernetes.io/projected/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-kube-api-access-msmbn\") pod \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") "
Oct 30 23:58:09.540301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.540288    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-openshift-ca\") pod \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") "
Oct 30 23:58:09.540423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.540405    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-xks-ca\") pod \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\" (UID: \"f9e4c4bf-0aa0-47c1-af55-ff8b73028512\") "
Oct 30 23:58:09.541028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.540985    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-openshift-ca" pod "f9e4c4bf-0aa0-47c1-af55-ff8b73028512" (UID: "f9e4c4bf-0aa0-47c1-af55-ff8b73028512"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:09.544427 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.544351    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-xks-ca" pod "f9e4c4bf-0aa0-47c1-af55-ff8b73028512" (UID: "f9e4c4bf-0aa0-47c1-af55-ff8b73028512"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:09.545036 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.545014    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-kube-api-access-msmbn" pod "f9e4c4bf-0aa0-47c1-af55-ff8b73028512" (UID: "f9e4c4bf-0aa0-47c1-af55-ff8b73028512"). InnerVolumeSpecName "kube-api-access-msmbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:09.643595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.643511    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:09.643595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.643538    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:09.643595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.643547    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msmbn\" (UniqueName: \"kubernetes.io/projected/f9e4c4bf-0aa0-47c1-af55-ff8b73028512-kube-api-access-msmbn\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:09.880388 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:09.880335    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"]
Oct 30 23:58:09.882772 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:58:09.882746    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca96124c_888e_4d50_b285_ef517527688a.slice/crio-434e6b2e50412ab082e79e409ff3236221ec0d8d54bb7ba990136e5e465c3b9d WatchSource:0}: Error finding container 434e6b2e50412ab082e79e409ff3236221ec0d8d54bb7ba990136e5e465c3b9d: Status 404 returned error can't find the container with id 434e6b2e50412ab082e79e409ff3236221ec0d8d54bb7ba990136e5e465c3b9d
Oct 30 23:58:10.162473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.160980    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"]
Oct 30 23:58:10.162473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.162103    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerName="revoke-keys"
Oct 30 23:58:10.162473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.162124    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerName="revoke-keys"
Oct 30 23:58:10.162473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.162137    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerName="revoke-keys"
Oct 30 23:58:10.162473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.162143    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerName="revoke-keys"
Oct 30 23:58:10.162473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.162361    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerName="revoke-keys"
Oct 30 23:58:10.162473 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.162392    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerName="revoke-keys"
Oct 30 23:58:10.176390 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.167946    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.227168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.227056    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk" event={"ID":"f9e4c4bf-0aa0-47c1-af55-ff8b73028512","Type":"ContainerDied","Data":"4f6a4656a9df188866c85780c30cf7de3b5c1c0d3c52342fdf0747ff5bc1298e"}
Oct 30 23:58:10.227168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.227140    4555 scope.go:122] "RemoveContainer" containerID="9c0381d2a95cc08a73db53907aa0c2e013f545d2bf2a91f4a5ac6b8ee3c154a5"
Oct 30 23:58:10.228757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.228689    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"
Oct 30 23:58:10.235102 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.235072    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w" event={"ID":"ca96124c-888e-4d50-b285-ef517527688a","Type":"ContainerStarted","Data":"434e6b2e50412ab082e79e409ff3236221ec0d8d54bb7ba990136e5e465c3b9d"}
Oct 30 23:58:10.242551 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.242529    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"]
Oct 30 23:58:10.265039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.265005    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88093177-87qbj\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.265173 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.265074    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88093177-87qbj\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.265414 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.265360    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzcw\" (UniqueName: \"kubernetes.io/projected/10c96fc4-d681-4a0e-beb4-290bd5756ed4-kube-api-access-7zzcw\") pod \"maas-api-revoke-sub-88093177-87qbj\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.281742 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.281664    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"]
Oct 30 23:58:10.289413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.289383    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-6x9kk"]
Oct 30 23:58:10.375430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.369879    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzcw\" (UniqueName: \"kubernetes.io/projected/10c96fc4-d681-4a0e-beb4-290bd5756ed4-kube-api-access-7zzcw\") pod \"maas-api-revoke-sub-88093177-87qbj\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.375430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.370011    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88093177-87qbj\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.375430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.370059    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88093177-87qbj\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.375430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.370887    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88093177-87qbj\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.375430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.375412    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88093177-87qbj\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.387257 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.387225    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzcw\" (UniqueName: \"kubernetes.io/projected/10c96fc4-d681-4a0e-beb4-290bd5756ed4-kube-api-access-7zzcw\") pod \"maas-api-revoke-sub-88093177-87qbj\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.540258 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.539952    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:10.838947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:10.838917    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"]
Oct 30 23:58:10.839830 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:58:10.839797    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c96fc4_d681_4a0e_beb4_290bd5756ed4.slice/crio-6fe50697a6703e85ae2fb524b5c7be593c045b265f72b71f572a6a22cb935c24 WatchSource:0}: Error finding container 6fe50697a6703e85ae2fb524b5c7be593c045b265f72b71f572a6a22cb935c24: Status 404 returned error can't find the container with id 6fe50697a6703e85ae2fb524b5c7be593c045b265f72b71f572a6a22cb935c24
Oct 30 23:58:11.156187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:11.156083    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"]
Oct 30 23:58:11.165900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:11.157567    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh" podUID="78ae695d-9222-4a0a-900b-441c8486efbc" containerName="maas-api" containerID="cri-o://3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46" gracePeriod=30
Oct 30 23:58:11.222996 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:11.222958    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" path="/var/lib/kubelet/pods/f9e4c4bf-0aa0-47c1-af55-ff8b73028512/volumes"
Oct 30 23:58:11.246413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:11.246332    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w" event={"ID":"ca96124c-888e-4d50-b285-ef517527688a","Type":"ContainerStarted","Data":"79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064"}
Oct 30 23:58:11.246602 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:11.246437    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:11.250103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:11.250078    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj" event={"ID":"10c96fc4-d681-4a0e-beb4-290bd5756ed4","Type":"ContainerStarted","Data":"6fe50697a6703e85ae2fb524b5c7be593c045b265f72b71f572a6a22cb935c24"}
Oct 30 23:58:11.290292 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:11.290220    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w" podStartSLOduration=1.845907266 podStartE2EDuration="2.290199781s" totalImagesPullingTime="444.292515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:11.286393272 +0000 UTC m=+2350.213030860" watchObservedRunningTime="2026-10-30 23:58:11.290199781 +0000 UTC m=+2350.216837363"
Oct 30 23:58:12.035518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.035461    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"]
Oct 30 23:58:12.042942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.042909    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-23f75e-8wsz7"]
Oct 30 23:58:12.118151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.118117    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:58:12.259921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.259886    4555 generic.go:356] "Generic (PLEG): container finished" podID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerID="7c662df93976b84896526aa423da357bab828dc15792b4a002b11548ccce3259" exitCode=7
Oct 30 23:58:12.260385 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.259979    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj" event={"ID":"10c96fc4-d681-4a0e-beb4-290bd5756ed4","Type":"ContainerDied","Data":"7c662df93976b84896526aa423da357bab828dc15792b4a002b11548ccce3259"}
Oct 30 23:58:12.260385 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.260345    4555 scope.go:122] "RemoveContainer" containerID="7c662df93976b84896526aa423da357bab828dc15792b4a002b11548ccce3259"
Oct 30 23:58:12.261734 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.261707    4555 generic.go:356] "Generic (PLEG): container finished" podID="78ae695d-9222-4a0a-900b-441c8486efbc" containerID="3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46" exitCode=0
Oct 30 23:58:12.261837 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.261760    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"
Oct 30 23:58:12.261837 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.261790    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh" event={"ID":"78ae695d-9222-4a0a-900b-441c8486efbc","Type":"ContainerDied","Data":"3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46"}
Oct 30 23:58:12.261954 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.261849    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh" event={"ID":"78ae695d-9222-4a0a-900b-441c8486efbc","Type":"ContainerDied","Data":"2e1cc64a415b223114f2c5a1dead8331976a83480401966bd44254d711d21e92"}
Oct 30 23:58:12.261954 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.261878    4555 scope.go:122] "RemoveContainer" containerID="3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46"
Oct 30 23:58:12.265987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.265960    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ae695d-9222-4a0a-900b-441c8486efbc-kube-api-access-5smld\" (UniqueName: \"kubernetes.io/projected/78ae695d-9222-4a0a-900b-441c8486efbc-kube-api-access-5smld\") pod \"78ae695d-9222-4a0a-900b-441c8486efbc\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") "
Oct 30 23:58:12.266093 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.266047    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-maas-api-tls\") pod \"78ae695d-9222-4a0a-900b-441c8486efbc\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") "
Oct 30 23:58:12.266261 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.266241    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-metrics-certs\") pod \"78ae695d-9222-4a0a-900b-441c8486efbc\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") "
Oct 30 23:58:12.266405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.266349    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ae695d-9222-4a0a-900b-441c8486efbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ae695d-9222-4a0a-900b-441c8486efbc-openshift-service-ca\") pod \"78ae695d-9222-4a0a-900b-441c8486efbc\" (UID: \"78ae695d-9222-4a0a-900b-441c8486efbc\") "
Oct 30 23:58:12.285174 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.285078    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ae695d-9222-4a0a-900b-441c8486efbc-openshift-service-ca" pod "78ae695d-9222-4a0a-900b-441c8486efbc" (UID: "78ae695d-9222-4a0a-900b-441c8486efbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:12.286724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.286695    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ae695d-9222-4a0a-900b-441c8486efbc-kube-api-access-5smld" pod "78ae695d-9222-4a0a-900b-441c8486efbc" (UID: "78ae695d-9222-4a0a-900b-441c8486efbc"). InnerVolumeSpecName "kube-api-access-5smld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:12.287222 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.287202    4555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ae695d-9222-4a0a-900b-441c8486efbc-openshift-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:12.287394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.287355    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5smld\" (UniqueName: \"kubernetes.io/projected/78ae695d-9222-4a0a-900b-441c8486efbc-kube-api-access-5smld\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:12.287870 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.287841    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-maas-api-tls" pod "78ae695d-9222-4a0a-900b-441c8486efbc" (UID: "78ae695d-9222-4a0a-900b-441c8486efbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:12.297692 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.297660    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-metrics-certs" pod "78ae695d-9222-4a0a-900b-441c8486efbc" (UID: "78ae695d-9222-4a0a-900b-441c8486efbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:12.309541 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.309488    4555 scope.go:122] "RemoveContainer" containerID="3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46"
Oct 30 23:58:12.309962 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:12.309925    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46\": container with ID starting with 3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46 not found: ID does not exist" containerID="3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46"
Oct 30 23:58:12.310087 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.309975    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46"} err="failed to get container status \"3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46\": rpc error: code = NotFound desc = could not find container \"3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46\": container with ID starting with 3af2270b6213315c67088020ac70af33051156632cc55a65387418cda062bf46 not found: ID does not exist"
Oct 30 23:58:12.389618 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.389589    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-maas-api-tls\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:12.389618 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.389618    4555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78ae695d-9222-4a0a-900b-441c8486efbc-metrics-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:12.602488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.602461    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"]
Oct 30 23:58:12.607559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:12.607530    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh"]
Oct 30 23:58:13.213445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:13.213406    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ae695d-9222-4a0a-900b-441c8486efbc" path="/var/lib/kubelet/pods/78ae695d-9222-4a0a-900b-441c8486efbc/volumes"
Oct 30 23:58:13.213893 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:13.213878    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a016483e-936f-45f5-a6c8-62bc13386af1" path="/var/lib/kubelet/pods/a016483e-936f-45f5-a6c8-62bc13386af1/volumes"
Oct 30 23:58:13.272166 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:13.272130    4555 generic.go:356] "Generic (PLEG): container finished" podID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerID="a6e8ddc2c9119943c0434af85bc7c568d32ae3d763753186820a358d7ff9cf77" exitCode=7
Oct 30 23:58:13.272644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:13.272220    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj" event={"ID":"10c96fc4-d681-4a0e-beb4-290bd5756ed4","Type":"ContainerDied","Data":"a6e8ddc2c9119943c0434af85bc7c568d32ae3d763753186820a358d7ff9cf77"}
Oct 30 23:58:13.272644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:13.272277    4555 scope.go:122] "RemoveContainer" containerID="7c662df93976b84896526aa423da357bab828dc15792b4a002b11548ccce3259"
Oct 30 23:58:13.272765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:13.272703    4555 scope.go:122] "RemoveContainer" containerID="a6e8ddc2c9119943c0434af85bc7c568d32ae3d763753186820a358d7ff9cf77"
Oct 30 23:58:13.273181 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:13.273154    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-88093177-87qbj_odh-ai-gateway-infra(10c96fc4-d681-4a0e-beb4-290bd5756ed4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4"
Oct 30 23:58:14.309987 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:14.305909    4555 scope.go:122] "RemoveContainer" containerID="a6e8ddc2c9119943c0434af85bc7c568d32ae3d763753186820a358d7ff9cf77"
Oct 30 23:58:14.309987 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:14.306979    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-88093177-87qbj_odh-ai-gateway-infra(10c96fc4-d681-4a0e-beb4-290bd5756ed4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4"
Oct 30 23:58:14.582508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:14.582406    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-23f75e-6b8bdf4d85-hcksh" podUID="78ae695d-9222-4a0a-900b-441c8486efbc" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.143:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:58:17.090053 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:17.090017    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"]
Oct 30 23:58:17.268097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:17.268065    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 30 23:58:18.489215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.489173    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"]
Oct 30 23:58:18.489916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.489900    4555 cpu_manager.go:414] "containerMap: removing container" podUID="78ae695d-9222-4a0a-900b-441c8486efbc" containerName="maas-api"
Oct 30 23:58:18.489916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.489916    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ae695d-9222-4a0a-900b-441c8486efbc" containerName="maas-api"
Oct 30 23:58:18.490021 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.489956    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerName="revoke-keys"
Oct 30 23:58:18.490021 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.489961    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerName="revoke-keys"
Oct 30 23:58:18.490086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.490064    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ae695d-9222-4a0a-900b-441c8486efbc" containerName="maas-api"
Oct 30 23:58:18.490086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.490079    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e4c4bf-0aa0-47c1-af55-ff8b73028512" containerName="revoke-keys"
Oct 30 23:58:18.495192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.495177    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:18.503355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.503328    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"]
Oct 30 23:58:18.595438 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.595406    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-tczdv\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:18.595753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.595727    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmcmm\" (UniqueName: \"kubernetes.io/projected/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-kube-api-access-pmcmm\") pod \"maas-api-revoke-sub-f993bcbd-tczdv\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:18.595937 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.595922    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-tczdv\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:18.697768 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.697735    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-tczdv\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:18.697934 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.697800    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-tczdv\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:18.697977 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.697954    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmcmm\" (UniqueName: \"kubernetes.io/projected/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-kube-api-access-pmcmm\") pod \"maas-api-revoke-sub-f993bcbd-tczdv\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:18.698582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.698561    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-tczdv\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:18.700423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.700395    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-tczdv\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:18.710444 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.710418    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmcmm\" (UniqueName: \"kubernetes.io/projected/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-kube-api-access-pmcmm\") pod \"maas-api-revoke-sub-f993bcbd-tczdv\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:18.815459 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:18.815429    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:19.111718 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:58:19.111677    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13dee1a_e4ce_458a_a3e9_730a8d9f1881.slice/crio-386b9b3d563b3a00096be4ab51dde52b5d4783a25e6a74418e8fde73753d4e31 WatchSource:0}: Error finding container 386b9b3d563b3a00096be4ab51dde52b5d4783a25e6a74418e8fde73753d4e31: Status 404 returned error can't find the container with id 386b9b3d563b3a00096be4ab51dde52b5d4783a25e6a74418e8fde73753d4e31
Oct 30 23:58:19.116537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:19.115283    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"]
Oct 30 23:58:19.200885 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:19.200309    4555 scope.go:122] "RemoveContainer" containerID="7f1ec5aaa7c1ec624571e01bc1b9d5dfafe63ec7b8d9f29754969ecba2d91ceb"
Oct 30 23:58:19.409595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:19.409560    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv" event={"ID":"a13dee1a-e4ce-458a-a3e9-730a8d9f1881","Type":"ContainerStarted","Data":"386b9b3d563b3a00096be4ab51dde52b5d4783a25e6a74418e8fde73753d4e31"}
Oct 30 23:58:20.420471 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:20.420357    4555 generic.go:356] "Generic (PLEG): container finished" podID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerID="305f14ceca67ba593c1540cd3d1646adb9e8097112476295d5c3f05103bb3cd3" exitCode=7
Oct 30 23:58:20.420471 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:20.420441    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv" event={"ID":"a13dee1a-e4ce-458a-a3e9-730a8d9f1881","Type":"ContainerDied","Data":"305f14ceca67ba593c1540cd3d1646adb9e8097112476295d5c3f05103bb3cd3"}
Oct 30 23:58:20.421007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:20.420866    4555 scope.go:122] "RemoveContainer" containerID="305f14ceca67ba593c1540cd3d1646adb9e8097112476295d5c3f05103bb3cd3"
Oct 30 23:58:20.422852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:20.422826    4555 generic.go:356] "Generic (PLEG): container finished" podID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerID="60e9f0ccaf61658bb4e2211ba7d116f59f749866c31d22ae4bd5ef8d1b8179c1" exitCode=7
Oct 30 23:58:20.423042 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:20.422900    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4" event={"ID":"66366b53-e15d-4e4e-959b-f5b3a6adf586","Type":"ContainerDied","Data":"60e9f0ccaf61658bb4e2211ba7d116f59f749866c31d22ae4bd5ef8d1b8179c1"}
Oct 30 23:58:20.423042 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:20.422961    4555 scope.go:122] "RemoveContainer" containerID="7f1ec5aaa7c1ec624571e01bc1b9d5dfafe63ec7b8d9f29754969ecba2d91ceb"
Oct 30 23:58:20.423475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:20.423450    4555 scope.go:122] "RemoveContainer" containerID="60e9f0ccaf61658bb4e2211ba7d116f59f749866c31d22ae4bd5ef8d1b8179c1"
Oct 30 23:58:20.423817 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:20.423790    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-cd8m4_odh-ai-gateway-infra(66366b53-e15d-4e4e-959b-f5b3a6adf586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586"
Oct 30 23:58:21.432628 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.432593    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv" event={"ID":"a13dee1a-e4ce-458a-a3e9-730a8d9f1881","Type":"ContainerStarted","Data":"3675a8b7bf655a73360d8a7b05bee79cdc27e72d274200ebff733ba3240cadd5"}
Oct 30 23:58:21.460382 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.460325    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv" podStartSLOduration=3.460311703 podStartE2EDuration="3.460311703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:21.459305114 +0000 UTC m=+2360.385942696" watchObservedRunningTime="2026-10-30 23:58:21.460311703 +0000 UTC m=+2360.386949283"
Oct 30 23:58:21.483839 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.483576    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"]
Oct 30 23:58:21.740951 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.740924    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:21.875052 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.875019    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66366b53-e15d-4e4e-959b-f5b3a6adf586-kube-api-access-6l6x7\" (UniqueName: \"kubernetes.io/projected/66366b53-e15d-4e4e-959b-f5b3a6adf586-kube-api-access-6l6x7\") pod \"66366b53-e15d-4e4e-959b-f5b3a6adf586\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") "
Oct 30 23:58:21.875360 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.875338    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-openshift-ca\") pod \"66366b53-e15d-4e4e-959b-f5b3a6adf586\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") "
Oct 30 23:58:21.875477 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.875456    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-xks-ca\") pod \"66366b53-e15d-4e4e-959b-f5b3a6adf586\" (UID: \"66366b53-e15d-4e4e-959b-f5b3a6adf586\") "
Oct 30 23:58:21.875703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.875676    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-openshift-ca" pod "66366b53-e15d-4e4e-959b-f5b3a6adf586" (UID: "66366b53-e15d-4e4e-959b-f5b3a6adf586"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:21.876125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.876107    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:21.877552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.877521    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-xks-ca" pod "66366b53-e15d-4e4e-959b-f5b3a6adf586" (UID: "66366b53-e15d-4e4e-959b-f5b3a6adf586"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:21.877788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.877768    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66366b53-e15d-4e4e-959b-f5b3a6adf586-kube-api-access-6l6x7" pod "66366b53-e15d-4e4e-959b-f5b3a6adf586" (UID: "66366b53-e15d-4e4e-959b-f5b3a6adf586"). InnerVolumeSpecName "kube-api-access-6l6x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:21.977551 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.977466    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66366b53-e15d-4e4e-959b-f5b3a6adf586-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:21.977551 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:21.977498    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l6x7\" (UniqueName: \"kubernetes.io/projected/66366b53-e15d-4e4e-959b-f5b3a6adf586-kube-api-access-6l6x7\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:22.444713 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:22.444673    4555 generic.go:356] "Generic (PLEG): container finished" podID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerID="3675a8b7bf655a73360d8a7b05bee79cdc27e72d274200ebff733ba3240cadd5" exitCode=7
Oct 30 23:58:22.445177 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:22.444825    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv" event={"ID":"a13dee1a-e4ce-458a-a3e9-730a8d9f1881","Type":"ContainerDied","Data":"3675a8b7bf655a73360d8a7b05bee79cdc27e72d274200ebff733ba3240cadd5"}
Oct 30 23:58:22.445177 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:22.444897    4555 scope.go:122] "RemoveContainer" containerID="305f14ceca67ba593c1540cd3d1646adb9e8097112476295d5c3f05103bb3cd3"
Oct 30 23:58:22.445310 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:22.445179    4555 scope.go:122] "RemoveContainer" containerID="3675a8b7bf655a73360d8a7b05bee79cdc27e72d274200ebff733ba3240cadd5"
Oct 30 23:58:22.445645 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:22.445604    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f993bcbd-tczdv_odh-ai-gateway-infra(a13dee1a-e4ce-458a-a3e9-730a8d9f1881)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881"
Oct 30 23:58:22.446589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:22.446566    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"
Oct 30 23:58:22.446763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:22.446589    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4" event={"ID":"66366b53-e15d-4e4e-959b-f5b3a6adf586","Type":"ContainerDied","Data":"282aa9f47b3d1873868203fdf814a26481de89dc922bb0c78faf497f52dfe967"}
Oct 30 23:58:22.465457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:22.465441    4555 scope.go:122] "RemoveContainer" containerID="60e9f0ccaf61658bb4e2211ba7d116f59f749866c31d22ae4bd5ef8d1b8179c1"
Oct 30 23:58:22.597311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:22.597278    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"]
Oct 30 23:58:22.620244 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:22.620214    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-cd8m4"]
Oct 30 23:58:23.211259 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:23.211223    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586" path="/var/lib/kubelet/pods/66366b53-e15d-4e4e-959b-f5b3a6adf586/volumes"
Oct 30 23:58:23.457157 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:23.457125    4555 scope.go:122] "RemoveContainer" containerID="3675a8b7bf655a73360d8a7b05bee79cdc27e72d274200ebff733ba3240cadd5"
Oct 30 23:58:23.457696 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:23.457668    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f993bcbd-tczdv_odh-ai-gateway-infra(a13dee1a-e4ce-458a-a3e9-730a8d9f1881)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881"
Oct 30 23:58:24.788803 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:24.788755    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7845b665bc-52x8q" podUID="a6b71183-9683-421d-84bd-bb1225ada1a9" containerName="console" containerID="cri-o://9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e" gracePeriod=15
Oct 30 23:58:25.148426 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.148400    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:58:25.200051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.200018    4555 scope.go:122] "RemoveContainer" containerID="a6e8ddc2c9119943c0434af85bc7c568d32ae3d763753186820a358d7ff9cf77"
Oct 30 23:58:25.228181 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.228150    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b71183-9683-421d-84bd-bb1225ada1a9-kube-api-access-plxhm\" (UniqueName: \"kubernetes.io/projected/a6b71183-9683-421d-84bd-bb1225ada1a9-kube-api-access-plxhm\") pod \"a6b71183-9683-421d-84bd-bb1225ada1a9\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") "
Oct 30 23:58:25.228429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.228210    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b71183-9683-421d-84bd-bb1225ada1a9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6b71183-9683-421d-84bd-bb1225ada1a9-tmp\") pod \"a6b71183-9683-421d-84bd-bb1225ada1a9\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") "
Oct 30 23:58:25.228429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.228260    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-serving-cert\") pod \"a6b71183-9683-421d-84bd-bb1225ada1a9\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") "
Oct 30 23:58:25.228429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.228408    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-oauth-config\") pod \"a6b71183-9683-421d-84bd-bb1225ada1a9\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") "
Oct 30 23:58:25.228705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.228451    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-service-ca\") pod \"a6b71183-9683-421d-84bd-bb1225ada1a9\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") "
Oct 30 23:58:25.228705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.228480    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-oauth-serving-cert\") pod \"a6b71183-9683-421d-84bd-bb1225ada1a9\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") "
Oct 30 23:58:25.228705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.228667    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-trusted-ca-bundle\") pod \"a6b71183-9683-421d-84bd-bb1225ada1a9\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") "
Oct 30 23:58:25.228874 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.228708    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-console-config\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-console-config\") pod \"a6b71183-9683-421d-84bd-bb1225ada1a9\" (UID: \"a6b71183-9683-421d-84bd-bb1225ada1a9\") "
Oct 30 23:58:25.229812 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.229779    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b71183-9683-421d-84bd-bb1225ada1a9-tmp" pod "a6b71183-9683-421d-84bd-bb1225ada1a9" (UID: "a6b71183-9683-421d-84bd-bb1225ada1a9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:25.230011 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.229921    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-console-config" pod "a6b71183-9683-421d-84bd-bb1225ada1a9" (UID: "a6b71183-9683-421d-84bd-bb1225ada1a9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:25.230222 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.230193    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-service-ca" pod "a6b71183-9683-421d-84bd-bb1225ada1a9" (UID: "a6b71183-9683-421d-84bd-bb1225ada1a9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:25.230657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.230630    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-trusted-ca-bundle" pod "a6b71183-9683-421d-84bd-bb1225ada1a9" (UID: "a6b71183-9683-421d-84bd-bb1225ada1a9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:25.235719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.233472    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-oauth-serving-cert" pod "a6b71183-9683-421d-84bd-bb1225ada1a9" (UID: "a6b71183-9683-421d-84bd-bb1225ada1a9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:25.239452 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.239428    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-oauth-config" pod "a6b71183-9683-421d-84bd-bb1225ada1a9" (UID: "a6b71183-9683-421d-84bd-bb1225ada1a9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:25.239805 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.239725    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b71183-9683-421d-84bd-bb1225ada1a9-kube-api-access-plxhm" pod "a6b71183-9683-421d-84bd-bb1225ada1a9" (UID: "a6b71183-9683-421d-84bd-bb1225ada1a9"). InnerVolumeSpecName "kube-api-access-plxhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:25.240759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.240736    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-serving-cert" pod "a6b71183-9683-421d-84bd-bb1225ada1a9" (UID: "a6b71183-9683-421d-84bd-bb1225ada1a9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:25.331832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.331802    4555 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6b71183-9683-421d-84bd-bb1225ada1a9-tmp\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:25.331832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.331835    4555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-serving-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:25.332073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.331850    4555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6b71183-9683-421d-84bd-bb1225ada1a9-console-oauth-config\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:25.332073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.331864    4555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:25.332073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.331878    4555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-oauth-serving-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:25.332073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.331891    4555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-trusted-ca-bundle\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:25.332073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.331904    4555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6b71183-9683-421d-84bd-bb1225ada1a9-console-config\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:25.332073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.331917    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plxhm\" (UniqueName: \"kubernetes.io/projected/a6b71183-9683-421d-84bd-bb1225ada1a9-kube-api-access-plxhm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:25.478999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.478895    4555 generic.go:356] "Generic (PLEG): container finished" podID="a6b71183-9683-421d-84bd-bb1225ada1a9" containerID="9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e" exitCode=0
Oct 30 23:58:25.478999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.478958    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7845b665bc-52x8q"
Oct 30 23:58:25.478999 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.478983    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7845b665bc-52x8q" event={"ID":"a6b71183-9683-421d-84bd-bb1225ada1a9","Type":"ContainerDied","Data":"9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e"}
Oct 30 23:58:25.479268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.479049    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7845b665bc-52x8q" event={"ID":"a6b71183-9683-421d-84bd-bb1225ada1a9","Type":"ContainerDied","Data":"8fd409414c03671798741ffde74e2cade25291f94eafe8cfb58e94187932404b"}
Oct 30 23:58:25.479268 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.479079    4555 scope.go:122] "RemoveContainer" containerID="9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e"
Oct 30 23:58:25.491383 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.491348    4555 scope.go:122] "RemoveContainer" containerID="9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e"
Oct 30 23:58:25.491734 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:25.491713    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e\": container with ID starting with 9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e not found: ID does not exist" containerID="9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e"
Oct 30 23:58:25.491811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.491741    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e"} err="failed to get container status \"9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e\": rpc error: code = NotFound desc = could not find container \"9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e\": container with ID starting with 9268aff0803e8b982cca2ab578ef90a988eb33c8c94f8421ad4801200222b35e not found: ID does not exist"
Oct 30 23:58:25.514981 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.514953    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7845b665bc-52x8q"]
Oct 30 23:58:25.519830 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:25.519803    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7845b665bc-52x8q"]
Oct 30 23:58:26.489825 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:26.489749    4555 generic.go:356] "Generic (PLEG): container finished" podID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerID="47d48547259f5acdf8455f69b2c8fd2609fd7bcb230c634ced137577dd7088f7" exitCode=7
Oct 30 23:58:26.490253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:26.489822    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj" event={"ID":"10c96fc4-d681-4a0e-beb4-290bd5756ed4","Type":"ContainerDied","Data":"47d48547259f5acdf8455f69b2c8fd2609fd7bcb230c634ced137577dd7088f7"}
Oct 30 23:58:26.490253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:26.489883    4555 scope.go:122] "RemoveContainer" containerID="a6e8ddc2c9119943c0434af85bc7c568d32ae3d763753186820a358d7ff9cf77"
Oct 30 23:58:26.490387 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:26.490347    4555 scope.go:122] "RemoveContainer" containerID="47d48547259f5acdf8455f69b2c8fd2609fd7bcb230c634ced137577dd7088f7"
Oct 30 23:58:26.490812 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:26.490785    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-88093177-87qbj_odh-ai-gateway-infra(10c96fc4-d681-4a0e-beb4-290bd5756ed4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4"
Oct 30 23:58:27.540359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.540077    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b71183-9683-421d-84bd-bb1225ada1a9" path="/var/lib/kubelet/pods/a6b71183-9683-421d-84bd-bb1225ada1a9/volumes"
Oct 30 23:58:27.578980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.578950    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"]
Oct 30 23:58:27.593648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.593615    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b0299-dfhjx"]
Oct 30 23:58:27.604327 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.603810    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"]
Oct 30 23:58:27.662513 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:27.662444    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:58:27.870835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.870807    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:27.992250 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.992212    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-openshift-ca\") pod \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") "
Oct 30 23:58:27.992486 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.992327    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-xks-ca\") pod \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") "
Oct 30 23:58:27.992486 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.992415    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c96fc4-d681-4a0e-beb4-290bd5756ed4-kube-api-access-7zzcw\" (UniqueName: \"kubernetes.io/projected/10c96fc4-d681-4a0e-beb4-290bd5756ed4-kube-api-access-7zzcw\") pod \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\" (UID: \"10c96fc4-d681-4a0e-beb4-290bd5756ed4\") "
Oct 30 23:58:27.992724 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.992698    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-openshift-ca" pod "10c96fc4-d681-4a0e-beb4-290bd5756ed4" (UID: "10c96fc4-d681-4a0e-beb4-290bd5756ed4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:27.993326 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.993299    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:27.994829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.994802    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-xks-ca" pod "10c96fc4-d681-4a0e-beb4-290bd5756ed4" (UID: "10c96fc4-d681-4a0e-beb4-290bd5756ed4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:27.994932 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:27.994839    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c96fc4-d681-4a0e-beb4-290bd5756ed4-kube-api-access-7zzcw" pod "10c96fc4-d681-4a0e-beb4-290bd5756ed4" (UID: "10c96fc4-d681-4a0e-beb4-290bd5756ed4"). InnerVolumeSpecName "kube-api-access-7zzcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:28.095743 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:28.095657    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c96fc4-d681-4a0e-beb4-290bd5756ed4-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:28.095963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:28.095946    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zzcw\" (UniqueName: \"kubernetes.io/projected/10c96fc4-d681-4a0e-beb4-290bd5756ed4-kube-api-access-7zzcw\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:28.595930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:28.595897    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"
Oct 30 23:58:28.595930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:28.595920    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj" event={"ID":"10c96fc4-d681-4a0e-beb4-290bd5756ed4","Type":"ContainerDied","Data":"6fe50697a6703e85ae2fb524b5c7be593c045b265f72b71f572a6a22cb935c24"}
Oct 30 23:58:28.596508 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:28.595993    4555 scope.go:122] "RemoveContainer" containerID="47d48547259f5acdf8455f69b2c8fd2609fd7bcb230c634ced137577dd7088f7"
Oct 30 23:58:28.627221 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:28.627193    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"]
Oct 30 23:58:28.643073 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:28.643048    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-87qbj"]
Oct 30 23:58:29.210301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:29.210261    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" path="/var/lib/kubelet/pods/10c96fc4-d681-4a0e-beb4-290bd5756ed4/volumes"
Oct 30 23:58:29.210860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:29.210842    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc137c6-08f7-4080-a0df-c22951462c12" path="/var/lib/kubelet/pods/3cc137c6-08f7-4080-a0df-c22951462c12/volumes"
Oct 30 23:58:30.206169 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.206131    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"]
Oct 30 23:58:30.207212 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207192    4555 cpu_manager.go:414] "containerMap: removing container" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerName="revoke-keys"
Oct 30 23:58:30.207281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207215    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerName="revoke-keys"
Oct 30 23:58:30.207281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207230    4555 cpu_manager.go:414] "containerMap: removing container" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerName="revoke-keys"
Oct 30 23:58:30.207281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207238    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerName="revoke-keys"
Oct 30 23:58:30.207281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207251    4555 cpu_manager.go:414] "containerMap: removing container" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerName="revoke-keys"
Oct 30 23:58:30.207281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207259    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerName="revoke-keys"
Oct 30 23:58:30.207476 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207306    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a6b71183-9683-421d-84bd-bb1225ada1a9" containerName="console"
Oct 30 23:58:30.207476 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207313    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b71183-9683-421d-84bd-bb1225ada1a9" containerName="console"
Oct 30 23:58:30.207476 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207336    4555 cpu_manager.go:414] "containerMap: removing container" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerName="revoke-keys"
Oct 30 23:58:30.207476 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207343    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerName="revoke-keys"
Oct 30 23:58:30.207476 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207360    4555 cpu_manager.go:414] "containerMap: removing container" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerName="revoke-keys"
Oct 30 23:58:30.207476 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207384    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerName="revoke-keys"
Oct 30 23:58:30.207669 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207569    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerName="revoke-keys"
Oct 30 23:58:30.207669 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207581    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerName="revoke-keys"
Oct 30 23:58:30.207669 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207592    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerName="revoke-keys"
Oct 30 23:58:30.207669 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207607    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerName="revoke-keys"
Oct 30 23:58:30.207669 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207618    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b71183-9683-421d-84bd-bb1225ada1a9" containerName="console"
Oct 30 23:58:30.207669 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.207631    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerName="revoke-keys"
Oct 30 23:58:30.213610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.213581    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.223123 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.222841    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"]
Oct 30 23:58:30.229776 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.229744    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxxrj\" (UniqueName: \"kubernetes.io/projected/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-kube-api-access-wxxrj\") pod \"maas-api-revoke-sub-88093177-lshsg\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.229928 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.229891    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88093177-lshsg\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.230007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.229933    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88093177-lshsg\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.332093 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.331953    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxxrj\" (UniqueName: \"kubernetes.io/projected/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-kube-api-access-wxxrj\") pod \"maas-api-revoke-sub-88093177-lshsg\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.333574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.333544    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88093177-lshsg\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.333805 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.333784    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88093177-lshsg\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.335235 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.335211    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88093177-lshsg\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.339937 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.339884    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88093177-lshsg\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.370435 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.363495    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxxrj\" (UniqueName: \"kubernetes.io/projected/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-kube-api-access-wxxrj\") pod \"maas-api-revoke-sub-88093177-lshsg\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.529980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.529945    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:30.841058 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:30.841029    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"]
Oct 30 23:58:30.841423 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:58:30.841397    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a0ee7f_8dc2_4728_be4e_47a518efb3cb.slice/crio-038fde8b215bbeab5b4188de8737d161d80c074404f1fb82670cfa78aa1c468f WatchSource:0}: Error finding container 038fde8b215bbeab5b4188de8737d161d80c074404f1fb82670cfa78aa1c468f: Status 404 returned error can't find the container with id 038fde8b215bbeab5b4188de8737d161d80c074404f1fb82670cfa78aa1c468f
Oct 30 23:58:31.639285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:31.639173    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg" event={"ID":"35a0ee7f-8dc2-4728-be4e-47a518efb3cb","Type":"ContainerStarted","Data":"210d08079b05ff3ac050861385a41607360a67c6d159a7e10cf295660922d9e4"}
Oct 30 23:58:31.639285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:31.639248    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg" event={"ID":"35a0ee7f-8dc2-4728-be4e-47a518efb3cb","Type":"ContainerStarted","Data":"038fde8b215bbeab5b4188de8737d161d80c074404f1fb82670cfa78aa1c468f"}
Oct 30 23:58:31.687395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:31.685879    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg" podStartSLOduration=1.685861228 podStartE2EDuration="1.685861228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:31.68073798 +0000 UTC m=+2370.607375564" watchObservedRunningTime="2026-10-30 23:58:31.685861228 +0000 UTC m=+2370.612498810"
Oct 30 23:58:32.112287 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.112250    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"]
Oct 30 23:58:32.113055 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.113036    4555 cpu_manager.go:414] "containerMap: removing container" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerName="revoke-keys"
Oct 30 23:58:32.113105 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.113059    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66366b53-e15d-4e4e-959b-f5b3a6adf586" containerName="revoke-keys"
Oct 30 23:58:32.113255 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.113244    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c96fc4-d681-4a0e-beb4-290bd5756ed4" containerName="revoke-keys"
Oct 30 23:58:32.117024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.116974    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" containerName="main" containerID="cri-o://6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a" gracePeriod=45
Oct 30 23:58:32.117196 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.117176    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.138513 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.138479    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"]
Oct 30 23:58:32.193072 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.193040    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8hb5\" (UniqueName: \"kubernetes.io/projected/10e708eb-84f4-491e-9e41-0805b6f6659f-kube-api-access-q8hb5\") pod \"maas-api-revoke-sub-91354e50-5jbqb\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.193252 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.193184    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-5jbqb\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.193484 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.193467    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-5jbqb\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.295109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.295071    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hb5\" (UniqueName: \"kubernetes.io/projected/10e708eb-84f4-491e-9e41-0805b6f6659f-kube-api-access-q8hb5\") pod \"maas-api-revoke-sub-91354e50-5jbqb\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.295281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.295193    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-5jbqb\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.295449 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.295433    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-5jbqb\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.296269 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.296230    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91354e50-5jbqb\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.298041 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.298016    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91354e50-5jbqb\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.323654 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.323627    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hb5\" (UniqueName: \"kubernetes.io/projected/10e708eb-84f4-491e-9e41-0805b6f6659f-kube-api-access-q8hb5\") pod \"maas-api-revoke-sub-91354e50-5jbqb\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.428683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.428654    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:58:32.452096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.452069    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:32.503603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.501590    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-tmp-dir\") pod \"5a721584-d8fe-4b76-b899-774cb70760d8\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") "
Oct 30 23:58:32.503603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.501680    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-home\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-home\") pod \"5a721584-d8fe-4b76-b899-774cb70760d8\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") "
Oct 30 23:58:32.503603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.501777    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-dshm\") pod \"5a721584-d8fe-4b76-b899-774cb70760d8\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") "
Oct 30 23:58:32.503603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.501874    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a721584-d8fe-4b76-b899-774cb70760d8-kube-api-access-c9slr\" (UniqueName: \"kubernetes.io/projected/5a721584-d8fe-4b76-b899-774cb70760d8-kube-api-access-c9slr\") pod \"5a721584-d8fe-4b76-b899-774cb70760d8\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") "
Oct 30 23:58:32.503603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.501917    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a721584-d8fe-4b76-b899-774cb70760d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a721584-d8fe-4b76-b899-774cb70760d8-tls-certs\") pod \"5a721584-d8fe-4b76-b899-774cb70760d8\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") "
Oct 30 23:58:32.503603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.502176    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-model-cache\") pod \"5a721584-d8fe-4b76-b899-774cb70760d8\" (UID: \"5a721584-d8fe-4b76-b899-774cb70760d8\") "
Oct 30 23:58:32.503603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.503131    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-model-cache" pod "5a721584-d8fe-4b76-b899-774cb70760d8" (UID: "5a721584-d8fe-4b76-b899-774cb70760d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:32.503603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.503324    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-tmp-dir" pod "5a721584-d8fe-4b76-b899-774cb70760d8" (UID: "5a721584-d8fe-4b76-b899-774cb70760d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:32.503603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.503560    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-home" pod "5a721584-d8fe-4b76-b899-774cb70760d8" (UID: "5a721584-d8fe-4b76-b899-774cb70760d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:32.529908 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.519513    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-dshm" pod "5a721584-d8fe-4b76-b899-774cb70760d8" (UID: "5a721584-d8fe-4b76-b899-774cb70760d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:32.529908 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.521846    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a721584-d8fe-4b76-b899-774cb70760d8-kube-api-access-c9slr" pod "5a721584-d8fe-4b76-b899-774cb70760d8" (UID: "5a721584-d8fe-4b76-b899-774cb70760d8"). InnerVolumeSpecName "kube-api-access-c9slr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:32.534396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.532736    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a721584-d8fe-4b76-b899-774cb70760d8-tls-certs" pod "5a721584-d8fe-4b76-b899-774cb70760d8" (UID: "5a721584-d8fe-4b76-b899-774cb70760d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:32.612050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.611695    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:32.612050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.611730    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:32.612050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.611746    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:32.612050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.611759    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a721584-d8fe-4b76-b899-774cb70760d8-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:32.612050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.611773    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9slr\" (UniqueName: \"kubernetes.io/projected/5a721584-d8fe-4b76-b899-774cb70760d8-kube-api-access-c9slr\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:32.612050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.611787    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a721584-d8fe-4b76-b899-774cb70760d8-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:32.651067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.650970    4555 generic.go:356] "Generic (PLEG): container finished" podID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerID="210d08079b05ff3ac050861385a41607360a67c6d159a7e10cf295660922d9e4" exitCode=7
Oct 30 23:58:32.651552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.651062    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg" event={"ID":"35a0ee7f-8dc2-4728-be4e-47a518efb3cb","Type":"ContainerDied","Data":"210d08079b05ff3ac050861385a41607360a67c6d159a7e10cf295660922d9e4"}
Oct 30 23:58:32.651693 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.651556    4555 scope.go:122] "RemoveContainer" containerID="210d08079b05ff3ac050861385a41607360a67c6d159a7e10cf295660922d9e4"
Oct 30 23:58:32.653114 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.653087    4555 generic.go:356] "Generic (PLEG): container finished" podID="5a721584-d8fe-4b76-b899-774cb70760d8" containerID="6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a" exitCode=0
Oct 30 23:58:32.653260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.653151    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"
Oct 30 23:58:32.653260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.653186    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" event={"ID":"5a721584-d8fe-4b76-b899-774cb70760d8","Type":"ContainerDied","Data":"6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a"}
Oct 30 23:58:32.653260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.653219    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p" event={"ID":"5a721584-d8fe-4b76-b899-774cb70760d8","Type":"ContainerDied","Data":"ddf21b7bda35688a33f16874915c5355945730a946e8b64ed809c64c7bced4ec"}
Oct 30 23:58:32.653260 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.653242    4555 scope.go:122] "RemoveContainer" containerID="6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a"
Oct 30 23:58:32.677357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.677279    4555 scope.go:122] "RemoveContainer" containerID="6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a"
Oct 30 23:58:32.677874 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:32.677843    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a\": container with ID starting with 6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a not found: ID does not exist" containerID="6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a"
Oct 30 23:58:32.677976 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.677887    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a"} err="failed to get container status \"6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a\": rpc error: code = NotFound desc = could not find container \"6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a\": container with ID starting with 6addf69544678ae864f388ca707335a9272aa1c56f3bcaefe5dffc29030e0c3a not found: ID does not exist"
Oct 30 23:58:32.730108 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.730080    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"]
Oct 30 23:58:32.744009 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.743983    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-23f75e/test-model-23f75e-kserve-74fc74f67f-hlh9p"]
Oct 30 23:58:32.831170 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:32.831127    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"]
Oct 30 23:58:32.838741 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:58:32.838121    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e708eb_84f4_491e_9e41_0805b6f6659f.slice/crio-ef7e73d01890e4952e03cf349486641952e27921f5d85c8db551e6afe129daa6 WatchSource:0}: Error finding container ef7e73d01890e4952e03cf349486641952e27921f5d85c8db551e6afe129daa6: Status 404 returned error can't find the container with id ef7e73d01890e4952e03cf349486641952e27921f5d85c8db551e6afe129daa6
Oct 30 23:58:33.038620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:33.038585    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"]
Oct 30 23:58:33.210846 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:33.210816    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" path="/var/lib/kubelet/pods/5a721584-d8fe-4b76-b899-774cb70760d8/volumes"
Oct 30 23:58:33.668104 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:33.668002    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg" event={"ID":"35a0ee7f-8dc2-4728-be4e-47a518efb3cb","Type":"ContainerStarted","Data":"d7236a3af76d5d8359c1016168fa08fcbe0e202f9c085ce66fd20e7a552a7322"}
Oct 30 23:58:33.670694 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:33.670663    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" event={"ID":"10e708eb-84f4-491e-9e41-0805b6f6659f","Type":"ContainerStarted","Data":"1585ac02054eb1caeeb17d36c29624e6b2c81abf605732189cf2b9a0ddde42d0"}
Oct 30 23:58:33.670826 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:33.670726    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" event={"ID":"10e708eb-84f4-491e-9e41-0805b6f6659f","Type":"ContainerStarted","Data":"ef7e73d01890e4952e03cf349486641952e27921f5d85c8db551e6afe129daa6"}
Oct 30 23:58:33.750787 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:33.750734    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" podStartSLOduration=1.750717162 podStartE2EDuration="1.750717162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:33.748725095 +0000 UTC m=+2372.675362676" watchObservedRunningTime="2026-10-30 23:58:33.750717162 +0000 UTC m=+2372.677354744"
Oct 30 23:58:34.686869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:34.686831    4555 generic.go:356] "Generic (PLEG): container finished" podID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerID="d7236a3af76d5d8359c1016168fa08fcbe0e202f9c085ce66fd20e7a552a7322" exitCode=7
Oct 30 23:58:34.687402 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:34.686953    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg" event={"ID":"35a0ee7f-8dc2-4728-be4e-47a518efb3cb","Type":"ContainerDied","Data":"d7236a3af76d5d8359c1016168fa08fcbe0e202f9c085ce66fd20e7a552a7322"}
Oct 30 23:58:34.687472 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:34.687403    4555 scope.go:122] "RemoveContainer" containerID="210d08079b05ff3ac050861385a41607360a67c6d159a7e10cf295660922d9e4"
Oct 30 23:58:34.690308 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:34.690267    4555 scope.go:122] "RemoveContainer" containerID="d7236a3af76d5d8359c1016168fa08fcbe0e202f9c085ce66fd20e7a552a7322"
Oct 30 23:58:34.690834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:34.690810    4555 generic.go:356] "Generic (PLEG): container finished" podID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerID="1585ac02054eb1caeeb17d36c29624e6b2c81abf605732189cf2b9a0ddde42d0" exitCode=7
Oct 30 23:58:34.690970 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:34.690951    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" event={"ID":"10e708eb-84f4-491e-9e41-0805b6f6659f","Type":"ContainerDied","Data":"1585ac02054eb1caeeb17d36c29624e6b2c81abf605732189cf2b9a0ddde42d0"}
Oct 30 23:58:34.691767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:34.691683    4555 scope.go:122] "RemoveContainer" containerID="1585ac02054eb1caeeb17d36c29624e6b2c81abf605732189cf2b9a0ddde42d0"
Oct 30 23:58:34.692896 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:34.692518    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-88093177-lshsg_odh-ai-gateway-infra(35a0ee7f-8dc2-4728-be4e-47a518efb3cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb"
Oct 30 23:58:35.200516 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:35.200481    4555 scope.go:122] "RemoveContainer" containerID="3675a8b7bf655a73360d8a7b05bee79cdc27e72d274200ebff733ba3240cadd5"
Oct 30 23:58:35.772447 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:35.770036    4555 scope.go:122] "RemoveContainer" containerID="d7236a3af76d5d8359c1016168fa08fcbe0e202f9c085ce66fd20e7a552a7322"
Oct 30 23:58:35.772447 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:35.770509    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-88093177-lshsg_odh-ai-gateway-infra(35a0ee7f-8dc2-4728-be4e-47a518efb3cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb"
Oct 30 23:58:35.790787 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:35.789296    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" event={"ID":"10e708eb-84f4-491e-9e41-0805b6f6659f","Type":"ContainerStarted","Data":"6981ebcb1c50fa720cdb43895e9f7859e2d6f2690fed9e38ae20973a9949e479"}
Oct 30 23:58:35.790787 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:35.789835    4555 scope.go:122] "RemoveContainer" containerID="6981ebcb1c50fa720cdb43895e9f7859e2d6f2690fed9e38ae20973a9949e479"
Oct 30 23:58:35.790787 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:35.790230    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-5jbqb_odh-ai-gateway-infra(10e708eb-84f4-491e-9e41-0805b6f6659f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f"
Oct 30 23:58:36.805269 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:36.805231    4555 generic.go:356] "Generic (PLEG): container finished" podID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerID="4d02b709883115714189c31524527878a08995296aa318080658a198da735f42" exitCode=7
Oct 30 23:58:36.805799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:36.805297    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv" event={"ID":"a13dee1a-e4ce-458a-a3e9-730a8d9f1881","Type":"ContainerDied","Data":"4d02b709883115714189c31524527878a08995296aa318080658a198da735f42"}
Oct 30 23:58:36.805799 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:36.805360    4555 scope.go:122] "RemoveContainer" containerID="3675a8b7bf655a73360d8a7b05bee79cdc27e72d274200ebff733ba3240cadd5"
Oct 30 23:58:36.805961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:36.805823    4555 scope.go:122] "RemoveContainer" containerID="4d02b709883115714189c31524527878a08995296aa318080658a198da735f42"
Oct 30 23:58:36.806234 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:36.806204    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f993bcbd-tczdv_odh-ai-gateway-infra(a13dee1a-e4ce-458a-a3e9-730a8d9f1881)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881"
Oct 30 23:58:36.808009 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:36.807972    4555 generic.go:356] "Generic (PLEG): container finished" podID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerID="6981ebcb1c50fa720cdb43895e9f7859e2d6f2690fed9e38ae20973a9949e479" exitCode=7
Oct 30 23:58:36.808125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:36.808067    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" event={"ID":"10e708eb-84f4-491e-9e41-0805b6f6659f","Type":"ContainerDied","Data":"6981ebcb1c50fa720cdb43895e9f7859e2d6f2690fed9e38ae20973a9949e479"}
Oct 30 23:58:36.808336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:36.808322    4555 scope.go:122] "RemoveContainer" containerID="6981ebcb1c50fa720cdb43895e9f7859e2d6f2690fed9e38ae20973a9949e479"
Oct 30 23:58:36.808681 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:36.808663    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-5jbqb_odh-ai-gateway-infra(10e708eb-84f4-491e-9e41-0805b6f6659f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f"
Oct 30 23:58:36.820716 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:36.820701    4555 scope.go:122] "RemoveContainer" containerID="1585ac02054eb1caeeb17d36c29624e6b2c81abf605732189cf2b9a0ddde42d0"
Oct 30 23:58:37.820104 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:37.820070    4555 scope.go:122] "RemoveContainer" containerID="6981ebcb1c50fa720cdb43895e9f7859e2d6f2690fed9e38ae20973a9949e479"
Oct 30 23:58:37.820537 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:37.820461    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-91354e50-5jbqb_odh-ai-gateway-infra(10e708eb-84f4-491e-9e41-0805b6f6659f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f"
Oct 30 23:58:37.890847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:37.890811    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"]
Oct 30 23:58:38.076961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.076881    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:38.128024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.127988    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-openshift-ca\") pod \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") "
Oct 30 23:58:38.128197 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.128123    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-kube-api-access-pmcmm\" (UniqueName: \"kubernetes.io/projected/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-kube-api-access-pmcmm\") pod \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") "
Oct 30 23:58:38.128197 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.128170    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-xks-ca\") pod \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\" (UID: \"a13dee1a-e4ce-458a-a3e9-730a8d9f1881\") "
Oct 30 23:58:38.129006 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.128931    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-openshift-ca" pod "a13dee1a-e4ce-458a-a3e9-730a8d9f1881" (UID: "a13dee1a-e4ce-458a-a3e9-730a8d9f1881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:38.129265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.129098    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:38.139839 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.139806    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-xks-ca" pod "a13dee1a-e4ce-458a-a3e9-730a8d9f1881" (UID: "a13dee1a-e4ce-458a-a3e9-730a8d9f1881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:38.139982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.139957    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-kube-api-access-pmcmm" pod "a13dee1a-e4ce-458a-a3e9-730a8d9f1881" (UID: "a13dee1a-e4ce-458a-a3e9-730a8d9f1881"). InnerVolumeSpecName "kube-api-access-pmcmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:38.230234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.230201    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmcmm\" (UniqueName: \"kubernetes.io/projected/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-kube-api-access-pmcmm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:38.230234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.230231    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13dee1a-e4ce-458a-a3e9-730a8d9f1881-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:38.829943 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.829903    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"
Oct 30 23:58:38.830439 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.829952    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv" event={"ID":"a13dee1a-e4ce-458a-a3e9-730a8d9f1881","Type":"ContainerDied","Data":"386b9b3d563b3a00096be4ab51dde52b5d4783a25e6a74418e8fde73753d4e31"}
Oct 30 23:58:38.830439 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.830030    4555 scope.go:122] "RemoveContainer" containerID="4d02b709883115714189c31524527878a08995296aa318080658a198da735f42"
Oct 30 23:58:38.916014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.915972    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"]
Oct 30 23:58:38.921561 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:38.921520    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tczdv"]
Oct 30 23:58:39.210258 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:39.210169    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" path="/var/lib/kubelet/pods/a13dee1a-e4ce-458a-a3e9-730a8d9f1881/volumes"
Oct 30 23:58:42.931356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.931267    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"]
Oct 30 23:58:42.932271 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.932246    4555 cpu_manager.go:414] "containerMap: removing container" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" containerName="main"
Oct 30 23:58:42.932420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.932272    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" containerName="main"
Oct 30 23:58:42.932420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.932293    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerName="revoke-keys"
Oct 30 23:58:42.932420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.932300    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerName="revoke-keys"
Oct 30 23:58:42.932420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.932325    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerName="revoke-keys"
Oct 30 23:58:42.932420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.932333    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerName="revoke-keys"
Oct 30 23:58:42.932716 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.932577    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerName="revoke-keys"
Oct 30 23:58:42.932716 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.932601    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerName="revoke-keys"
Oct 30 23:58:42.932716 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.932616    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a721584-d8fe-4b76-b899-774cb70760d8" containerName="main"
Oct 30 23:58:42.937705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.937682    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:42.948804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:42.948777    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"]
Oct 30 23:58:43.018497 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.018452    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cln7\" (UniqueName: \"kubernetes.io/projected/b1b52db4-a62f-45ce-8630-75172184ec77-kube-api-access-7cln7\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:43.018672 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.018569    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:43.018672 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.018629    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:43.120046 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.120008    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cln7\" (UniqueName: \"kubernetes.io/projected/b1b52db4-a62f-45ce-8630-75172184ec77-kube-api-access-7cln7\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:43.120265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.120065    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:43.120265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.120110    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:43.120792 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.120767    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:43.122985 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.122953    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:43.134913 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.134887    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cln7\" (UniqueName: \"kubernetes.io/projected/b1b52db4-a62f-45ce-8630-75172184ec77-kube-api-access-7cln7\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:43.254983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.254901    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:43.620599 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:58:43.620570    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b52db4_a62f_45ce_8630_75172184ec77.slice/crio-941f156d6d83f5449e3d0c449e5c1f69d17fbef3038c15839df52226b9b5f241 WatchSource:0}: Error finding container 941f156d6d83f5449e3d0c449e5c1f69d17fbef3038c15839df52226b9b5f241: Status 404 returned error can't find the container with id 941f156d6d83f5449e3d0c449e5c1f69d17fbef3038c15839df52226b9b5f241
Oct 30 23:58:43.621637 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.621604    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"]
Oct 30 23:58:43.874411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:43.874315    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r" event={"ID":"b1b52db4-a62f-45ce-8630-75172184ec77","Type":"ContainerStarted","Data":"941f156d6d83f5449e3d0c449e5c1f69d17fbef3038c15839df52226b9b5f241"}
Oct 30 23:58:44.885243 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:44.885143    4555 generic.go:356] "Generic (PLEG): container finished" podID="b1b52db4-a62f-45ce-8630-75172184ec77" containerID="f3bf028131c1b0763ea6a4f660e2305c58aa546d18f09afa75f0be3b60fe5c6a" exitCode=0
Oct 30 23:58:44.885749 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:44.885238    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r" event={"ID":"b1b52db4-a62f-45ce-8630-75172184ec77","Type":"ContainerDied","Data":"f3bf028131c1b0763ea6a4f660e2305c58aa546d18f09afa75f0be3b60fe5c6a"}
Oct 30 23:58:45.219445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:45.217141    4555 scope.go:122] "RemoveContainer" containerID="d7236a3af76d5d8359c1016168fa08fcbe0e202f9c085ce66fd20e7a552a7322"
Oct 30 23:58:46.119005 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.118967    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:46.210098 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.210063    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-openshift-ca\") pod \"b1b52db4-a62f-45ce-8630-75172184ec77\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") "
Oct 30 23:58:46.210446 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.210421    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b52db4-a62f-45ce-8630-75172184ec77-kube-api-access-7cln7\" (UniqueName: \"kubernetes.io/projected/b1b52db4-a62f-45ce-8630-75172184ec77-kube-api-access-7cln7\") pod \"b1b52db4-a62f-45ce-8630-75172184ec77\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") "
Oct 30 23:58:46.210577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.210421    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-openshift-ca" pod "b1b52db4-a62f-45ce-8630-75172184ec77" (UID: "b1b52db4-a62f-45ce-8630-75172184ec77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:46.210577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.210513    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-xks-ca\") pod \"b1b52db4-a62f-45ce-8630-75172184ec77\" (UID: \"b1b52db4-a62f-45ce-8630-75172184ec77\") "
Oct 30 23:58:46.211266 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.211237    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:46.213802 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.213781    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b52db4-a62f-45ce-8630-75172184ec77-kube-api-access-7cln7" pod "b1b52db4-a62f-45ce-8630-75172184ec77" (UID: "b1b52db4-a62f-45ce-8630-75172184ec77"). InnerVolumeSpecName "kube-api-access-7cln7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:46.213928 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.213848    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-xks-ca" pod "b1b52db4-a62f-45ce-8630-75172184ec77" (UID: "b1b52db4-a62f-45ce-8630-75172184ec77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:46.312930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.312891    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b52db4-a62f-45ce-8630-75172184ec77-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:46.312930 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.312924    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cln7\" (UniqueName: \"kubernetes.io/projected/b1b52db4-a62f-45ce-8630-75172184ec77-kube-api-access-7cln7\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:46.948424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.934610    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r" event={"ID":"b1b52db4-a62f-45ce-8630-75172184ec77","Type":"ContainerDied","Data":"941f156d6d83f5449e3d0c449e5c1f69d17fbef3038c15839df52226b9b5f241"}
Oct 30 23:58:46.948424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.934713    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941f156d6d83f5449e3d0c449e5c1f69d17fbef3038c15839df52226b9b5f241"
Oct 30 23:58:46.948424 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.934780    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"
Oct 30 23:58:46.978400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.978281    4555 generic.go:356] "Generic (PLEG): container finished" podID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerID="4fa928608c79a9243d4b24dd142f17ef7e3152aba1ed8cfc0884ae027c834f87" exitCode=0
Oct 30 23:58:46.978571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.978483    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg" event={"ID":"35a0ee7f-8dc2-4728-be4e-47a518efb3cb","Type":"ContainerDied","Data":"4fa928608c79a9243d4b24dd142f17ef7e3152aba1ed8cfc0884ae027c834f87"}
Oct 30 23:58:46.978571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:46.978539    4555 scope.go:122] "RemoveContainer" containerID="d7236a3af76d5d8359c1016168fa08fcbe0e202f9c085ce66fd20e7a552a7322"
Oct 30 23:58:47.204983 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:47.201828    4555 scope.go:122] "RemoveContainer" containerID="6981ebcb1c50fa720cdb43895e9f7859e2d6f2690fed9e38ae20973a9949e479"
Oct 30 23:58:47.610893 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:47.609660    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:58:48.004135 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.002280    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" event={"ID":"10e708eb-84f4-491e-9e41-0805b6f6659f","Type":"ContainerStarted","Data":"22b4e9008fce0f4e6f6943bdaa10bd3c22497a2e109b016a0a75deae4101cee9"}
Oct 30 23:58:48.046246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.045262    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"]
Oct 30 23:58:48.081859 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.081820    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt" podUID="f832f680-ca9d-434b-ac35-3c9c47105f8e" containerName="main" containerID="cri-o://6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e" gracePeriod=45
Oct 30 23:58:48.310758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.310692    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:48.395180 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.395138    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-kube-api-access-wxxrj\" (UniqueName: \"kubernetes.io/projected/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-kube-api-access-wxxrj\") pod \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") "
Oct 30 23:58:48.395398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.395285    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-openshift-ca\") pod \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") "
Oct 30 23:58:48.395602 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.395576    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-xks-ca\") pod \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\" (UID: \"35a0ee7f-8dc2-4728-be4e-47a518efb3cb\") "
Oct 30 23:58:48.397148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.397087    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-openshift-ca" pod "35a0ee7f-8dc2-4728-be4e-47a518efb3cb" (UID: "35a0ee7f-8dc2-4728-be4e-47a518efb3cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:48.405343 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.404649    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-xks-ca" pod "35a0ee7f-8dc2-4728-be4e-47a518efb3cb" (UID: "35a0ee7f-8dc2-4728-be4e-47a518efb3cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:48.405343 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.404882    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-kube-api-access-wxxrj" pod "35a0ee7f-8dc2-4728-be4e-47a518efb3cb" (UID: "35a0ee7f-8dc2-4728-be4e-47a518efb3cb"). InnerVolumeSpecName "kube-api-access-wxxrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:48.447791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.447754    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:58:48.498170 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.498088    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:48.498170 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.498125    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxxrj\" (UniqueName: \"kubernetes.io/projected/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-kube-api-access-wxxrj\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:48.498170 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.498143    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a0ee7f-8dc2-4728-be4e-47a518efb3cb-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:48.599753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.599725    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-model-cache\") pod \"f832f680-ca9d-434b-ac35-3c9c47105f8e\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") "
Oct 30 23:58:48.599914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.599807    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f832f680-ca9d-434b-ac35-3c9c47105f8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f832f680-ca9d-434b-ac35-3c9c47105f8e-tls-certs\") pod \"f832f680-ca9d-434b-ac35-3c9c47105f8e\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") "
Oct 30 23:58:48.599914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.599871    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f832f680-ca9d-434b-ac35-3c9c47105f8e-kube-api-access-jmmqt\" (UniqueName: \"kubernetes.io/projected/f832f680-ca9d-434b-ac35-3c9c47105f8e-kube-api-access-jmmqt\") pod \"f832f680-ca9d-434b-ac35-3c9c47105f8e\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") "
Oct 30 23:58:48.599914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.599902    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-tmp-dir\") pod \"f832f680-ca9d-434b-ac35-3c9c47105f8e\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") "
Oct 30 23:58:48.600081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.599986    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-model-cache" pod "f832f680-ca9d-434b-ac35-3c9c47105f8e" (UID: "f832f680-ca9d-434b-ac35-3c9c47105f8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:48.600144 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.600080    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-dshm\") pod \"f832f680-ca9d-434b-ac35-3c9c47105f8e\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") "
Oct 30 23:58:48.600259 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.600237    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-home\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-home\") pod \"f832f680-ca9d-434b-ac35-3c9c47105f8e\" (UID: \"f832f680-ca9d-434b-ac35-3c9c47105f8e\") "
Oct 30 23:58:48.600259 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.600239    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-tmp-dir" pod "f832f680-ca9d-434b-ac35-3c9c47105f8e" (UID: "f832f680-ca9d-434b-ac35-3c9c47105f8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:48.600547 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.600527    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-home" pod "f832f680-ca9d-434b-ac35-3c9c47105f8e" (UID: "f832f680-ca9d-434b-ac35-3c9c47105f8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:48.601333 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.601309    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:48.601437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.601341    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:48.601437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.601357    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:48.602285 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.602260    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f832f680-ca9d-434b-ac35-3c9c47105f8e-kube-api-access-jmmqt" pod "f832f680-ca9d-434b-ac35-3c9c47105f8e" (UID: "f832f680-ca9d-434b-ac35-3c9c47105f8e"). InnerVolumeSpecName "kube-api-access-jmmqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:48.602428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.602280    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-dshm" pod "f832f680-ca9d-434b-ac35-3c9c47105f8e" (UID: "f832f680-ca9d-434b-ac35-3c9c47105f8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:48.602428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.602284    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f832f680-ca9d-434b-ac35-3c9c47105f8e-tls-certs" pod "f832f680-ca9d-434b-ac35-3c9c47105f8e" (UID: "f832f680-ca9d-434b-ac35-3c9c47105f8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:48.702958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.702922    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f832f680-ca9d-434b-ac35-3c9c47105f8e-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:48.702958 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.702961    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmmqt\" (UniqueName: \"kubernetes.io/projected/f832f680-ca9d-434b-ac35-3c9c47105f8e-kube-api-access-jmmqt\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:48.703215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.702976    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f832f680-ca9d-434b-ac35-3c9c47105f8e-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:48.713581 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.713545    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"]
Oct 30 23:58:48.714721 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714695    4555 cpu_manager.go:414] "containerMap: removing container" podUID="b1b52db4-a62f-45ce-8630-75172184ec77" containerName="revoke-keys"
Oct 30 23:58:48.714721 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714719    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b52db4-a62f-45ce-8630-75172184ec77" containerName="revoke-keys"
Oct 30 23:58:48.714910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714749    4555 cpu_manager.go:414] "containerMap: removing container" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerName="revoke-keys"
Oct 30 23:58:48.714910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714758    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerName="revoke-keys"
Oct 30 23:58:48.714910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714769    4555 cpu_manager.go:414] "containerMap: removing container" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerName="revoke-keys"
Oct 30 23:58:48.714910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714776    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerName="revoke-keys"
Oct 30 23:58:48.714910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714813    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerName="revoke-keys"
Oct 30 23:58:48.714910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714819    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerName="revoke-keys"
Oct 30 23:58:48.714910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714832    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f832f680-ca9d-434b-ac35-3c9c47105f8e" containerName="main"
Oct 30 23:58:48.714910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714839    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f832f680-ca9d-434b-ac35-3c9c47105f8e" containerName="main"
Oct 30 23:58:48.714910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714852    4555 cpu_manager.go:414] "containerMap: removing container" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerName="revoke-keys"
Oct 30 23:58:48.714910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.714859    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerName="revoke-keys"
Oct 30 23:58:48.715481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.715074    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b52db4-a62f-45ce-8630-75172184ec77" containerName="revoke-keys"
Oct 30 23:58:48.715481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.715090    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerName="revoke-keys"
Oct 30 23:58:48.715481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.715098    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerName="revoke-keys"
Oct 30 23:58:48.715481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.715114    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13dee1a-e4ce-458a-a3e9-730a8d9f1881" containerName="revoke-keys"
Oct 30 23:58:48.715481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.715123    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f832f680-ca9d-434b-ac35-3c9c47105f8e" containerName="main"
Oct 30 23:58:48.715481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.715132    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" containerName="revoke-keys"
Oct 30 23:58:48.719950 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.719929    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:48.720114 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.720094    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"]
Oct 30 23:58:48.724910 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.724850    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.735470 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.735445    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6b38a4\"/\"default-dockercfg-m4qp2\""
Oct 30 23:58:48.735608 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.735534    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6b38a4\"/\"openshift-service-ca.crt\""
Oct 30 23:58:48.735670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.735630    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6b38a4\"/\"kube-root-ca.crt\""
Oct 30 23:58:48.735771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.735751    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6b38a4\"/\"test-model-6b38a4-kserve-self-signed-certs\""
Oct 30 23:58:48.750015 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.749964    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"]
Oct 30 23:58:48.756646 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.756624    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"]
Oct 30 23:58:48.805514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.805477    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxd9b\" (UniqueName: \"kubernetes.io/projected/847ba777-6164-4b2d-ac74-a90ba8553acc-kube-api-access-xxd9b\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.805714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.805603    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-tmckl\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:48.805714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.805666    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-dshm\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.805860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.805798    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-home\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.805860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.805837    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ba777-6164-4b2d-ac74-a90ba8553acc-tls-certs\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.805981 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.805931    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-tmp-dir\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.805981 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.805956    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4p6\" (UniqueName: \"kubernetes.io/projected/bcf68bc5-624f-47d6-b182-852ef9f76756-kube-api-access-rt4p6\") pod \"maas-api-revoke-sub-f993bcbd-tmckl\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:48.806099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.806053    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-tmckl\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:48.806157 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.806134    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-model-cache\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.907748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.907702    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-tmp-dir\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.907952 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.907757    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4p6\" (UniqueName: \"kubernetes.io/projected/bcf68bc5-624f-47d6-b182-852ef9f76756-kube-api-access-rt4p6\") pod \"maas-api-revoke-sub-f993bcbd-tmckl\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:48.907952 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.907845    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-tmckl\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:48.907952 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.907913    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-model-cache\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.908125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.907997    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxd9b\" (UniqueName: \"kubernetes.io/projected/847ba777-6164-4b2d-ac74-a90ba8553acc-kube-api-access-xxd9b\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.908125 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.908110    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-tmckl\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:48.908247 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.908169    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-dshm\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.908247 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.908216    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-home\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.908247 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.908239    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ba777-6164-4b2d-ac74-a90ba8553acc-tls-certs\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.908489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.908461    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-model-cache\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.908743 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.908166    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-tmp-dir\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.908826 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.908763    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f993bcbd-tmckl\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:48.909024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.908998    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-home\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.911412 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.911363    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-dshm\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.911990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.911965    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ba777-6164-4b2d-ac74-a90ba8553acc-tls-certs\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:48.912804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.912779    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f993bcbd-tmckl\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:48.928558 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.928530    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4p6\" (UniqueName: \"kubernetes.io/projected/bcf68bc5-624f-47d6-b182-852ef9f76756-kube-api-access-rt4p6\") pod \"maas-api-revoke-sub-f993bcbd-tmckl\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:48.931118 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:48.931093    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxd9b\" (UniqueName: \"kubernetes.io/projected/847ba777-6164-4b2d-ac74-a90ba8553acc-kube-api-access-xxd9b\") pod \"test-model-6b38a4-kserve-ccfdc56dc-m6vrn\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") " pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:49.094396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.072262    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:49.098392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.095915    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:49.138440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.133604    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"]
Oct 30 23:58:49.317119 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.317079    4555 generic.go:356] "Generic (PLEG): container finished" podID="f832f680-ca9d-434b-ac35-3c9c47105f8e" containerID="6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e" exitCode=0
Oct 30 23:58:49.319882 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.319857    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"
Oct 30 23:58:49.336778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.336749    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"
Oct 30 23:58:49.363758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.363659    4555 generic.go:356] "Generic (PLEG): container finished" podID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerID="22b4e9008fce0f4e6f6943bdaa10bd3c22497a2e109b016a0a75deae4101cee9" exitCode=0
Oct 30 23:58:49.407755 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.401989    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt" event={"ID":"f832f680-ca9d-434b-ac35-3c9c47105f8e","Type":"ContainerDied","Data":"6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e"}
Oct 30 23:58:49.407755 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.402058    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt" event={"ID":"f832f680-ca9d-434b-ac35-3c9c47105f8e","Type":"ContainerDied","Data":"e14fe55361956b9b1ca50bf02ee43b11465585d3f2b05319f26f576ca82b72cb"}
Oct 30 23:58:49.407755 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.402079    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg" event={"ID":"35a0ee7f-8dc2-4728-be4e-47a518efb3cb","Type":"ContainerDied","Data":"038fde8b215bbeab5b4188de8737d161d80c074404f1fb82670cfa78aa1c468f"}
Oct 30 23:58:49.407755 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.402122    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="038fde8b215bbeab5b4188de8737d161d80c074404f1fb82670cfa78aa1c468f"
Oct 30 23:58:49.407755 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.402135    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" event={"ID":"10e708eb-84f4-491e-9e41-0805b6f6659f","Type":"ContainerDied","Data":"22b4e9008fce0f4e6f6943bdaa10bd3c22497a2e109b016a0a75deae4101cee9"}
Oct 30 23:58:49.407755 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.402180    4555 scope.go:122] "RemoveContainer" containerID="6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e"
Oct 30 23:58:49.476176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.476146    4555 scope.go:122] "RemoveContainer" containerID="6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e"
Oct 30 23:58:49.476940 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:49.476569    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e\": container with ID starting with 6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e not found: ID does not exist" containerID="6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e"
Oct 30 23:58:49.476940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.476610    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e"} err="failed to get container status \"6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e\": rpc error: code = NotFound desc = could not find container \"6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e\": container with ID starting with 6d1a452e8bba30a9bc2400526009496cc3d4a0231f22fb98c8a83b835674570e not found: ID does not exist"
Oct 30 23:58:49.476940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.476637    4555 scope.go:122] "RemoveContainer" containerID="6981ebcb1c50fa720cdb43895e9f7859e2d6f2690fed9e38ae20973a9949e479"
Oct 30 23:58:49.525667 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.525628    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"]
Oct 30 23:58:49.533051 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.533021    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-lshsg"]
Oct 30 23:58:49.544821 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.544788    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"]
Oct 30 23:58:49.555848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.555819    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7b0299/test-model-7b0299-kserve-64c4b49cd9-kkxmt"]
Oct 30 23:58:49.651231 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.651142    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"]
Oct 30 23:58:49.914809 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.914668    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"]
Oct 30 23:58:49.936980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:49.936938    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"]
Oct 30 23:58:50.108917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.108766    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:50.209104 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.190154    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"]
Oct 30 23:58:50.224809 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.224776    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e708eb-84f4-491e-9e41-0805b6f6659f-kube-api-access-q8hb5\" (UniqueName: \"kubernetes.io/projected/10e708eb-84f4-491e-9e41-0805b6f6659f-kube-api-access-q8hb5\") pod \"10e708eb-84f4-491e-9e41-0805b6f6659f\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") "
Oct 30 23:58:50.224988 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.224850    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-openshift-ca\") pod \"10e708eb-84f4-491e-9e41-0805b6f6659f\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") "
Oct 30 23:58:50.225103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.225082    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-xks-ca\") pod \"10e708eb-84f4-491e-9e41-0805b6f6659f\" (UID: \"10e708eb-84f4-491e-9e41-0805b6f6659f\") "
Oct 30 23:58:50.235149 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.228532    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-openshift-ca" pod "10e708eb-84f4-491e-9e41-0805b6f6659f" (UID: "10e708eb-84f4-491e-9e41-0805b6f6659f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:50.235149 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.232505    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-xks-ca" pod "10e708eb-84f4-491e-9e41-0805b6f6659f" (UID: "10e708eb-84f4-491e-9e41-0805b6f6659f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:50.235149 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.232625    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e708eb-84f4-491e-9e41-0805b6f6659f-kube-api-access-q8hb5" pod "10e708eb-84f4-491e-9e41-0805b6f6659f" (UID: "10e708eb-84f4-491e-9e41-0805b6f6659f"). InnerVolumeSpecName "kube-api-access-q8hb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:50.337410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.336049    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:50.337410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.336087    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8hb5\" (UniqueName: \"kubernetes.io/projected/10e708eb-84f4-491e-9e41-0805b6f6659f-kube-api-access-q8hb5\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:50.337410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.336108    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e708eb-84f4-491e-9e41-0805b6f6659f-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:50.501678 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.501500    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl" event={"ID":"bcf68bc5-624f-47d6-b182-852ef9f76756","Type":"ContainerStarted","Data":"d008d798f821a5647726b458afa7fea9adf6a0b37e0245f7807724c8d152f5cc"}
Oct 30 23:58:50.503662 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.503633    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"]
Oct 30 23:58:50.505026 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.505002    4555 cpu_manager.go:414] "containerMap: removing container" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerName="revoke-keys"
Oct 30 23:58:50.505190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.505177    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerName="revoke-keys"
Oct 30 23:58:50.505292 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.505282    4555 cpu_manager.go:414] "containerMap: removing container" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerName="revoke-keys"
Oct 30 23:58:50.505398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.505386    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerName="revoke-keys"
Oct 30 23:58:50.505777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.505761    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerName="revoke-keys"
Oct 30 23:58:50.505898 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.505888    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerName="revoke-keys"
Oct 30 23:58:50.506586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.505006    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"
Oct 30 23:58:50.513405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.513354    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:50.513811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.513218    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb" event={"ID":"10e708eb-84f4-491e-9e41-0805b6f6659f","Type":"ContainerDied","Data":"ef7e73d01890e4952e03cf349486641952e27921f5d85c8db551e6afe129daa6"}
Oct 30 23:58:50.514264 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.514154    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn" event={"ID":"847ba777-6164-4b2d-ac74-a90ba8553acc","Type":"ContainerStarted","Data":"5c753db5f663db1c4cc02d01f6682aacb5316c52528030d1d6245fe4b1b53b10"}
Oct 30 23:58:50.514264 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.514201    4555 scope.go:122] "RemoveContainer" containerID="22b4e9008fce0f4e6f6943bdaa10bd3c22497a2e109b016a0a75deae4101cee9"
Oct 30 23:58:50.518727 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.518700    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"]
Oct 30 23:58:50.597654 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.596741    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn" podStartSLOduration=2.596722384 podStartE2EDuration="2.596722384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:50.594894299 +0000 UTC m=+2389.521531873" watchObservedRunningTime="2026-10-30 23:58:50.596722384 +0000 UTC m=+2389.523359969"
Oct 30 23:58:50.634904 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.634874    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"]
Oct 30 23:58:50.640651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.640619    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91354e50-5jbqb"]
Oct 30 23:58:50.676648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.676614    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5ct\" (UniqueName: \"kubernetes.io/projected/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-kube-api-access-qj5ct\") pod \"maas-api-revoke-sub-88093177-h2c42\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:50.676862 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.676708    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88093177-h2c42\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:50.676862 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.676781    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88093177-h2c42\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:50.787794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.787754    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88093177-h2c42\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:50.788198 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.788167    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88093177-h2c42\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:50.788726 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.788702    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5ct\" (UniqueName: \"kubernetes.io/projected/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-kube-api-access-qj5ct\") pod \"maas-api-revoke-sub-88093177-h2c42\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:50.789763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.789738    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88093177-h2c42\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:50.811176 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.811103    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5ct\" (UniqueName: \"kubernetes.io/projected/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-kube-api-access-qj5ct\") pod \"maas-api-revoke-sub-88093177-h2c42\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:50.813146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.813089    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88093177-h2c42\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:50.851998 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:50.851960    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:51.096397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:51.096295    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:51.160747 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:51.160715    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"]
Oct 30 23:58:51.162483 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:58:51.162301    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda8f2b1_c6f2_49a4_b7c3_ae7c096885da.slice/crio-8e35e45cf558a8b07d6b16e8a81196e7fd66a83acc7b867c87b7c643008f135b WatchSource:0}: Error finding container 8e35e45cf558a8b07d6b16e8a81196e7fd66a83acc7b867c87b7c643008f135b: Status 404 returned error can't find the container with id 8e35e45cf558a8b07d6b16e8a81196e7fd66a83acc7b867c87b7c643008f135b
Oct 30 23:58:51.215343 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:51.215304    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f" path="/var/lib/kubelet/pods/10e708eb-84f4-491e-9e41-0805b6f6659f/volumes"
Oct 30 23:58:51.216024 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:51.216004    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a0ee7f-8dc2-4728-be4e-47a518efb3cb" path="/var/lib/kubelet/pods/35a0ee7f-8dc2-4728-be4e-47a518efb3cb/volumes"
Oct 30 23:58:51.216665 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:51.216642    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f832f680-ca9d-434b-ac35-3c9c47105f8e" path="/var/lib/kubelet/pods/f832f680-ca9d-434b-ac35-3c9c47105f8e/volumes"
Oct 30 23:58:51.525913 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:51.525865    4555 generic.go:356] "Generic (PLEG): container finished" podID="bcf68bc5-624f-47d6-b182-852ef9f76756" containerID="4e5e2082b32438d07b9f9bfb4f0930b81b4a25d4ea88345706393b070cc0dc70" exitCode=0
Oct 30 23:58:51.526515 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:51.525916    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl" event={"ID":"bcf68bc5-624f-47d6-b182-852ef9f76756","Type":"ContainerDied","Data":"4e5e2082b32438d07b9f9bfb4f0930b81b4a25d4ea88345706393b070cc0dc70"}
Oct 30 23:58:51.529793 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:51.529725    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42" event={"ID":"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da","Type":"ContainerStarted","Data":"8e35e45cf558a8b07d6b16e8a81196e7fd66a83acc7b867c87b7c643008f135b"}
Oct 30 23:58:51.531980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:51.531950    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn" event={"ID":"847ba777-6164-4b2d-ac74-a90ba8553acc","Type":"ContainerStarted","Data":"ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27"}
Oct 30 23:58:52.547640 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:52.547607    4555 generic.go:356] "Generic (PLEG): container finished" podID="bda8f2b1-c6f2-49a4-b7c3-ae7c096885da" containerID="19a281b3c0ef6db3c536db71531dae407d71100f37874e86157d9ecd01b813f2" exitCode=0
Oct 30 23:58:52.548041 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:52.547716    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42" event={"ID":"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da","Type":"ContainerDied","Data":"19a281b3c0ef6db3c536db71531dae407d71100f37874e86157d9ecd01b813f2"}
Oct 30 23:58:52.776873 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:52.776843    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:52.831534 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:52.831490    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-xks-ca\") pod \"bcf68bc5-624f-47d6-b182-852ef9f76756\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") "
Oct 30 23:58:52.831690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:52.831610    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf68bc5-624f-47d6-b182-852ef9f76756-kube-api-access-rt4p6\" (UniqueName: \"kubernetes.io/projected/bcf68bc5-624f-47d6-b182-852ef9f76756-kube-api-access-rt4p6\") pod \"bcf68bc5-624f-47d6-b182-852ef9f76756\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") "
Oct 30 23:58:52.831690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:52.831667    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-openshift-ca\") pod \"bcf68bc5-624f-47d6-b182-852ef9f76756\" (UID: \"bcf68bc5-624f-47d6-b182-852ef9f76756\") "
Oct 30 23:58:52.832152 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:52.832098    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-openshift-ca" pod "bcf68bc5-624f-47d6-b182-852ef9f76756" (UID: "bcf68bc5-624f-47d6-b182-852ef9f76756"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:52.832696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:52.832667    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:52.833905 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:52.833879    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-xks-ca" pod "bcf68bc5-624f-47d6-b182-852ef9f76756" (UID: "bcf68bc5-624f-47d6-b182-852ef9f76756"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:52.834568 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:52.834546    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf68bc5-624f-47d6-b182-852ef9f76756-kube-api-access-rt4p6" pod "bcf68bc5-624f-47d6-b182-852ef9f76756" (UID: "bcf68bc5-624f-47d6-b182-852ef9f76756"). InnerVolumeSpecName "kube-api-access-rt4p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:53.005443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.004909    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf68bc5-624f-47d6-b182-852ef9f76756-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:53.005443 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.004951    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt4p6\" (UniqueName: \"kubernetes.io/projected/bcf68bc5-624f-47d6-b182-852ef9f76756-kube-api-access-rt4p6\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:53.558830 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.558739    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"
Oct 30 23:58:53.558830 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.558779    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl" event={"ID":"bcf68bc5-624f-47d6-b182-852ef9f76756","Type":"ContainerDied","Data":"d008d798f821a5647726b458afa7fea9adf6a0b37e0245f7807724c8d152f5cc"}
Oct 30 23:58:53.559350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.558838    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d008d798f821a5647726b458afa7fea9adf6a0b37e0245f7807724c8d152f5cc"
Oct 30 23:58:53.617158 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.617106    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"]
Oct 30 23:58:53.637555 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.633842    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"]
Oct 30 23:58:53.856533 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.856496    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:53.880481 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.880440    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-kube-api-access-qj5ct\" (UniqueName: \"kubernetes.io/projected/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-kube-api-access-qj5ct\") pod \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") "
Oct 30 23:58:53.880661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.880552    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-openshift-ca\") pod \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") "
Oct 30 23:58:53.880752 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.880660    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-xks-ca\") pod \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\" (UID: \"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da\") "
Oct 30 23:58:53.881039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.881010    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-openshift-ca" pod "bda8f2b1-c6f2-49a4-b7c3-ae7c096885da" (UID: "bda8f2b1-c6f2-49a4-b7c3-ae7c096885da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:53.881429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.881408    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:53.883670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.883641    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-kube-api-access-qj5ct" pod "bda8f2b1-c6f2-49a4-b7c3-ae7c096885da" (UID: "bda8f2b1-c6f2-49a4-b7c3-ae7c096885da"). InnerVolumeSpecName "kube-api-access-qj5ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:53.883824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.883805    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-xks-ca" pod "bda8f2b1-c6f2-49a4-b7c3-ae7c096885da" (UID: "bda8f2b1-c6f2-49a4-b7c3-ae7c096885da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:53.985385 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.985339    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj5ct\" (UniqueName: \"kubernetes.io/projected/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-kube-api-access-qj5ct\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:53.985605 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:53.985394    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:54.575748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:54.575717    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"
Oct 30 23:58:54.576272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:54.575749    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42" event={"ID":"bda8f2b1-c6f2-49a4-b7c3-ae7c096885da","Type":"ContainerDied","Data":"8e35e45cf558a8b07d6b16e8a81196e7fd66a83acc7b867c87b7c643008f135b"}
Oct 30 23:58:54.576272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:54.575810    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e35e45cf558a8b07d6b16e8a81196e7fd66a83acc7b867c87b7c643008f135b"
Oct 30 23:58:57.449042 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:57.449006    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:58:58.637844 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:58.637801    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p" podUID="bc377667-f4cb-420e-867b-35f94720b4cd" containerName="payload-processing" containerID="cri-o://19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8" gracePeriod=25
Oct 30 23:58:58.640247 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:58.637569    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs" podUID="0b5181de-449e-444d-a038-3defcb855281" containerName="payload-pre-processing" containerID="cri-o://8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9" gracePeriod=25
Oct 30 23:58:59.004205 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.004171    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:58:59.096658 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.096343    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:59.101572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.101538    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b5181de-449e-444d-a038-3defcb855281-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5181de-449e-444d-a038-3defcb855281-plugins-config-volume\") pod \"0b5181de-449e-444d-a038-3defcb855281\" (UID: \"0b5181de-449e-444d-a038-3defcb855281\") "
Oct 30 23:58:59.101769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.101597    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5181de-449e-444d-a038-3defcb855281-kube-api-access-2sb49\" (UniqueName: \"kubernetes.io/projected/0b5181de-449e-444d-a038-3defcb855281-kube-api-access-2sb49\") pod \"0b5181de-449e-444d-a038-3defcb855281\" (UID: \"0b5181de-449e-444d-a038-3defcb855281\") "
Oct 30 23:58:59.103190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.103127    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5181de-449e-444d-a038-3defcb855281-plugins-config-volume" pod "0b5181de-449e-444d-a038-3defcb855281" (UID: "0b5181de-449e-444d-a038-3defcb855281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:59.107638 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.107605    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:58:59.123102 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.123062    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5181de-449e-444d-a038-3defcb855281-kube-api-access-2sb49" pod "0b5181de-449e-444d-a038-3defcb855281" (UID: "0b5181de-449e-444d-a038-3defcb855281"). InnerVolumeSpecName "kube-api-access-2sb49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:59.127310 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.127276    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:59.203648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.203618    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc377667-f4cb-420e-867b-35f94720b4cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc377667-f4cb-420e-867b-35f94720b4cd-plugins-config-volume\") pod \"bc377667-f4cb-420e-867b-35f94720b4cd\" (UID: \"bc377667-f4cb-420e-867b-35f94720b4cd\") "
Oct 30 23:58:59.203831 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.203716    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc377667-f4cb-420e-867b-35f94720b4cd-kube-api-access-c2wjt\" (UniqueName: \"kubernetes.io/projected/bc377667-f4cb-420e-867b-35f94720b4cd-kube-api-access-c2wjt\") pod \"bc377667-f4cb-420e-867b-35f94720b4cd\" (UID: \"bc377667-f4cb-420e-867b-35f94720b4cd\") "
Oct 30 23:58:59.204168 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.204101    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc377667-f4cb-420e-867b-35f94720b4cd-plugins-config-volume" pod "bc377667-f4cb-420e-867b-35f94720b4cd" (UID: "bc377667-f4cb-420e-867b-35f94720b4cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:59.205070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.204985    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5181de-449e-444d-a038-3defcb855281-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:59.205238 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.205213    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sb49\" (UniqueName: \"kubernetes.io/projected/0b5181de-449e-444d-a038-3defcb855281-kube-api-access-2sb49\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:59.205238 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.205235    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc377667-f4cb-420e-867b-35f94720b4cd-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:59.206677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.206651    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc377667-f4cb-420e-867b-35f94720b4cd-kube-api-access-c2wjt" pod "bc377667-f4cb-420e-867b-35f94720b4cd" (UID: "bc377667-f4cb-420e-867b-35f94720b4cd"). InnerVolumeSpecName "kube-api-access-c2wjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:59.307484 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.307330    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2wjt\" (UniqueName: \"kubernetes.io/projected/bc377667-f4cb-420e-867b-35f94720b4cd-kube-api-access-c2wjt\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:58:59.572869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.572809    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"]
Oct 30 23:58:59.573920 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.573890    4555 cpu_manager.go:414] "containerMap: removing container" podUID="bc377667-f4cb-420e-867b-35f94720b4cd" containerName="payload-processing"
Oct 30 23:58:59.574133 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.574115    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc377667-f4cb-420e-867b-35f94720b4cd" containerName="payload-processing"
Oct 30 23:58:59.574299 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.574285    4555 cpu_manager.go:414] "containerMap: removing container" podUID="bcf68bc5-624f-47d6-b182-852ef9f76756" containerName="revoke-keys"
Oct 30 23:58:59.574440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.574424    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf68bc5-624f-47d6-b182-852ef9f76756" containerName="revoke-keys"
Oct 30 23:58:59.574573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.574559    4555 cpu_manager.go:414] "containerMap: removing container" podUID="bda8f2b1-c6f2-49a4-b7c3-ae7c096885da" containerName="revoke-keys"
Oct 30 23:58:59.574705 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.574688    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda8f2b1-c6f2-49a4-b7c3-ae7c096885da" containerName="revoke-keys"
Oct 30 23:58:59.574851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.574839    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0b5181de-449e-444d-a038-3defcb855281" containerName="payload-pre-processing"
Oct 30 23:58:59.574929 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.574918    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5181de-449e-444d-a038-3defcb855281" containerName="payload-pre-processing"
Oct 30 23:58:59.575028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.575019    4555 cpu_manager.go:414] "containerMap: removing container" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerName="revoke-keys"
Oct 30 23:58:59.575099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.575090    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerName="revoke-keys"
Oct 30 23:58:59.575418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.575400    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e708eb-84f4-491e-9e41-0805b6f6659f" containerName="revoke-keys"
Oct 30 23:58:59.575540 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.575521    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5181de-449e-444d-a038-3defcb855281" containerName="payload-pre-processing"
Oct 30 23:58:59.575686 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.575554    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc377667-f4cb-420e-867b-35f94720b4cd" containerName="payload-processing"
Oct 30 23:58:59.575686 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.575572    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf68bc5-624f-47d6-b182-852ef9f76756" containerName="revoke-keys"
Oct 30 23:58:59.575686 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.575588    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda8f2b1-c6f2-49a4-b7c3-ae7c096885da" containerName="revoke-keys"
Oct 30 23:58:59.581155 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.580777    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:58:59.593631 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.593550    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"]
Oct 30 23:58:59.613940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.613545    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:58:59.613940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.613659    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48949\" (UniqueName: \"kubernetes.io/projected/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-kube-api-access-48949\") pod \"maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:58:59.613940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.613687    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:58:59.667709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.667680    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:58:59.668281 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.668002    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:58:59.681590 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.681556    4555 generic.go:356] "Generic (PLEG): container finished" podID="bc377667-f4cb-420e-867b-35f94720b4cd" containerID="19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8" exitCode=0
Oct 30 23:58:59.681766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.681597    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"
Oct 30 23:58:59.681766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.681634    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p" event={"ID":"bc377667-f4cb-420e-867b-35f94720b4cd","Type":"ContainerDied","Data":"19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8"}
Oct 30 23:58:59.681766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.681685    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p" event={"ID":"bc377667-f4cb-420e-867b-35f94720b4cd","Type":"ContainerDied","Data":"3c12428c7f90a8bce147dfe0f3eed0c3ada0d3101dfe79608294c0fa75a3c2ee"}
Oct 30 23:58:59.681766 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.681713    4555 scope.go:122] "RemoveContainer" containerID="19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8"
Oct 30 23:58:59.683916 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.683891    4555 generic.go:356] "Generic (PLEG): container finished" podID="0b5181de-449e-444d-a038-3defcb855281" containerID="8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9" exitCode=0
Oct 30 23:58:59.684062 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.683998    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs" event={"ID":"0b5181de-449e-444d-a038-3defcb855281","Type":"ContainerDied","Data":"8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9"}
Oct 30 23:58:59.684062 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.684049    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs" event={"ID":"0b5181de-449e-444d-a038-3defcb855281","Type":"ContainerDied","Data":"f755cf3bb230b10e9442e7836f539ed5c4cf5169e94dc55a14b1f1ba599a71b4"}
Oct 30 23:58:59.684832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.684813    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"
Oct 30 23:58:59.696489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.696466    4555 scope.go:122] "RemoveContainer" containerID="19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8"
Oct 30 23:58:59.696871 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:59.696843    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8\": container with ID starting with 19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8 not found: ID does not exist" containerID="19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8"
Oct 30 23:58:59.696948 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.696885    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8"} err="failed to get container status \"19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8\": rpc error: code = NotFound desc = could not find container \"19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8\": container with ID starting with 19b6ff94c81a821095b71c261d30a0fd0205e426858c10002b9bcd7eb80c4ac8 not found: ID does not exist"
Oct 30 23:58:59.696948 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.696910    4555 scope.go:122] "RemoveContainer" containerID="8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9"
Oct 30 23:58:59.701122 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.701096    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 30 23:58:59.710687 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.710666    4555 scope.go:122] "RemoveContainer" containerID="8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9"
Oct 30 23:58:59.711076 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:58:59.711051    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9\": container with ID starting with 8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9 not found: ID does not exist" containerID="8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9"
Oct 30 23:58:59.711170 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.711090    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9"} err="failed to get container status \"8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9\": rpc error: code = NotFound desc = could not find container \"8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9\": container with ID starting with 8643593cfd71004a861c41b3e60b01f4b512b654ba7d11d3f12eb2df205cdea9 not found: ID does not exist"
Oct 30 23:58:59.715080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.715056    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:58:59.715211 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.715195    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48949\" (UniqueName: \"kubernetes.io/projected/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-kube-api-access-48949\") pod \"maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:58:59.715292 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.715233    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:58:59.715603 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.715580    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"]
Oct 30 23:58:59.718250 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.716361    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:58:59.721277 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.721257    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:58:59.723105 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.723063    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c5084-75f85789b7-6xf5p"]
Oct 30 23:58:59.731557 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.731532    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48949\" (UniqueName: \"kubernetes.io/projected/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-kube-api-access-48949\") pod \"maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:58:59.746499 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.746432    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"]
Oct 30 23:58:59.756395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.753336    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c5084-dd658bc5-szvjs"]
Oct 30 23:58:59.905088 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:58:59.905010    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:59:00.216323 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:59:00.216296    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d54bb7_b0bc_4fd1_be96_9082e9dea41b.slice/crio-1280c81e527fe0197ecce1ae637c9fd2049f0ae0caaff876eec5bd27ebe8f910 WatchSource:0}: Error finding container 1280c81e527fe0197ecce1ae637c9fd2049f0ae0caaff876eec5bd27ebe8f910: Status 404 returned error can't find the container with id 1280c81e527fe0197ecce1ae637c9fd2049f0ae0caaff876eec5bd27ebe8f910
Oct 30 23:59:00.217988 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:00.217960    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"]
Oct 30 23:59:00.359257 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:00.359217    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"]
Oct 30 23:59:00.483254 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:00.483163    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"]
Oct 30 23:59:00.931319 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:00.931276    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm" event={"ID":"29d54bb7-b0bc-4fd1-be96-9082e9dea41b","Type":"ContainerStarted","Data":"1280c81e527fe0197ecce1ae637c9fd2049f0ae0caaff876eec5bd27ebe8f910"}
Oct 30 23:59:01.042076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:01.041937    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"]
Oct 30 23:59:01.213444 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:01.212582    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5181de-449e-444d-a038-3defcb855281" path="/var/lib/kubelet/pods/0b5181de-449e-444d-a038-3defcb855281/volumes"
Oct 30 23:59:01.213444 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:01.213151    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc377667-f4cb-420e-867b-35f94720b4cd" path="/var/lib/kubelet/pods/bc377667-f4cb-420e-867b-35f94720b4cd/volumes"
Oct 30 23:59:01.810395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:01.806485    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"]
Oct 30 23:59:02.175928 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.174627    4555 generic.go:356] "Generic (PLEG): container finished" podID="29d54bb7-b0bc-4fd1-be96-9082e9dea41b" containerID="55521c29ccdaaf33e653b4a2f779f649f35e4bdde2f546e83cbca44e81cc2f2e" exitCode=0
Oct 30 23:59:02.175928 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.174905    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm" event={"ID":"29d54bb7-b0bc-4fd1-be96-9082e9dea41b","Type":"ContainerDied","Data":"55521c29ccdaaf33e653b4a2f779f649f35e4bdde2f546e83cbca44e81cc2f2e"}
Oct 30 23:59:02.235394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.235237    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"]
Oct 30 23:59:02.247054 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.247028    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.249192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.249144    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"]
Oct 30 23:59:02.373393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.372461    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.373393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.372824    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.373393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.372861    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4gxx\" (UniqueName: \"kubernetes.io/projected/f6680280-8130-4a71-a114-7d1a971ba1fb-kube-api-access-n4gxx\") pod \"maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.474592 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.474491    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.474765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.474748    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.474806 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.474779    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gxx\" (UniqueName: \"kubernetes.io/projected/f6680280-8130-4a71-a114-7d1a971ba1fb-kube-api-access-n4gxx\") pod \"maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.475389 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.475345    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.480355 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.480323    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.490301 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.490255    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gxx\" (UniqueName: \"kubernetes.io/projected/f6680280-8130-4a71-a114-7d1a971ba1fb-kube-api-access-n4gxx\") pod \"maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.584935 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.584898    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:02.948517 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:02.948452    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"]
Oct 30 23:59:03.186279 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.186224    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744" event={"ID":"f6680280-8130-4a71-a114-7d1a971ba1fb","Type":"ContainerStarted","Data":"cee8c48312200f7917272b281c968421b59ff037fa05eed7558e6a343d3b7b46"}
Oct 30 23:59:03.373621 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.373594    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:59:03.509913 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.509833    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-openshift-ca\") pod \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") "
Oct 30 23:59:03.509913 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.509911    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-kube-api-access-48949\" (UniqueName: \"kubernetes.io/projected/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-kube-api-access-48949\") pod \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") "
Oct 30 23:59:03.510142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.509974    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-xks-ca\") pod \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\" (UID: \"29d54bb7-b0bc-4fd1-be96-9082e9dea41b\") "
Oct 30 23:59:03.510250 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.510221    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-openshift-ca" pod "29d54bb7-b0bc-4fd1-be96-9082e9dea41b" (UID: "29d54bb7-b0bc-4fd1-be96-9082e9dea41b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:03.510783 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.510762    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:03.512246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.512223    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-xks-ca" pod "29d54bb7-b0bc-4fd1-be96-9082e9dea41b" (UID: "29d54bb7-b0bc-4fd1-be96-9082e9dea41b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:03.512680 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.512650    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-kube-api-access-48949" pod "29d54bb7-b0bc-4fd1-be96-9082e9dea41b" (UID: "29d54bb7-b0bc-4fd1-be96-9082e9dea41b"). InnerVolumeSpecName "kube-api-access-48949". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:03.568463 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.568430    4555 scope.go:122] "RemoveContainer" containerID="931bf0030239938932709164c65d112c64cbf40729d145776a63ff4b7ce459a5"
Oct 30 23:59:03.613131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.613095    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48949\" (UniqueName: \"kubernetes.io/projected/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-kube-api-access-48949\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:03.613131 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:03.613130    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d54bb7-b0bc-4fd1-be96-9082e9dea41b-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:04.196589 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:04.196553    4555 generic.go:356] "Generic (PLEG): container finished" podID="f6680280-8130-4a71-a114-7d1a971ba1fb" containerID="bb43029c6b65eb341d308debe3fde347d349a2b40e70f334f750ce70a5c3a494" exitCode=0
Oct 30 23:59:04.197039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:04.196641    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744" event={"ID":"f6680280-8130-4a71-a114-7d1a971ba1fb","Type":"ContainerDied","Data":"bb43029c6b65eb341d308debe3fde347d349a2b40e70f334f750ce70a5c3a494"}
Oct 30 23:59:04.198383 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:04.198346    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm" event={"ID":"29d54bb7-b0bc-4fd1-be96-9082e9dea41b","Type":"ContainerDied","Data":"1280c81e527fe0197ecce1ae637c9fd2049f0ae0caaff876eec5bd27ebe8f910"}
Oct 30 23:59:04.198491 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:04.198396    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"
Oct 30 23:59:04.198491 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:04.198397    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1280c81e527fe0197ecce1ae637c9fd2049f0ae0caaff876eec5bd27ebe8f910"
Oct 30 23:59:04.721594 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:04.717936    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v" podUID="648f4553-fa26-444b-a2e4-a690b559990f" containerName="main" containerID="cri-o://829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d" gracePeriod=45
Oct 30 23:59:04.790031 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:04.789988    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v" podUID="648f4553-fa26-444b-a2e4-a690b559990f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": dial tcp 10.133.0.153:8000: connect: connection refused"
Oct 30 23:59:05.060505 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.059529    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:59:05.150026 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.149985    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648f4553-fa26-444b-a2e4-a690b559990f-tls-certs\" (UniqueName: \"kubernetes.io/secret/648f4553-fa26-444b-a2e4-a690b559990f-tls-certs\") pod \"648f4553-fa26-444b-a2e4-a690b559990f\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") "
Oct 30 23:59:05.150189 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.150163    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648f4553-fa26-444b-a2e4-a690b559990f-kube-api-access-brz44\" (UniqueName: \"kubernetes.io/projected/648f4553-fa26-444b-a2e4-a690b559990f-kube-api-access-brz44\") pod \"648f4553-fa26-444b-a2e4-a690b559990f\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") "
Oct 30 23:59:05.150273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.150205    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-dshm\") pod \"648f4553-fa26-444b-a2e4-a690b559990f\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") "
Oct 30 23:59:05.150273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.150254    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-tmp-dir\") pod \"648f4553-fa26-444b-a2e4-a690b559990f\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") "
Oct 30 23:59:05.150442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.150328    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-home\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-home\") pod \"648f4553-fa26-444b-a2e4-a690b559990f\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") "
Oct 30 23:59:05.150442 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.150398    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-model-cache\") pod \"648f4553-fa26-444b-a2e4-a690b559990f\" (UID: \"648f4553-fa26-444b-a2e4-a690b559990f\") "
Oct 30 23:59:05.151109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.151061    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-tmp-dir" pod "648f4553-fa26-444b-a2e4-a690b559990f" (UID: "648f4553-fa26-444b-a2e4-a690b559990f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:05.151109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.151068    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-model-cache" pod "648f4553-fa26-444b-a2e4-a690b559990f" (UID: "648f4553-fa26-444b-a2e4-a690b559990f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:05.151267 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.151121    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-home" pod "648f4553-fa26-444b-a2e4-a690b559990f" (UID: "648f4553-fa26-444b-a2e4-a690b559990f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:05.153180 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.153148    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648f4553-fa26-444b-a2e4-a690b559990f-tls-certs" pod "648f4553-fa26-444b-a2e4-a690b559990f" (UID: "648f4553-fa26-444b-a2e4-a690b559990f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:05.153321 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.153214    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-dshm" pod "648f4553-fa26-444b-a2e4-a690b559990f" (UID: "648f4553-fa26-444b-a2e4-a690b559990f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:05.153451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.153353    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648f4553-fa26-444b-a2e4-a690b559990f-kube-api-access-brz44" pod "648f4553-fa26-444b-a2e4-a690b559990f" (UID: "648f4553-fa26-444b-a2e4-a690b559990f"). InnerVolumeSpecName "kube-api-access-brz44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:05.211564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.211519    4555 generic.go:356] "Generic (PLEG): container finished" podID="648f4553-fa26-444b-a2e4-a690b559990f" containerID="829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d" exitCode=0
Oct 30 23:59:05.211992 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.211973    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"
Oct 30 23:59:05.213890 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.213853    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v" event={"ID":"648f4553-fa26-444b-a2e4-a690b559990f","Type":"ContainerDied","Data":"829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d"}
Oct 30 23:59:05.214014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.213908    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v" event={"ID":"648f4553-fa26-444b-a2e4-a690b559990f","Type":"ContainerDied","Data":"dfb3ca12153962ff0c1c0c9b90b53963c896348a92206f12faa3c03dc3ed3957"}
Oct 30 23:59:05.214014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.213934    4555 scope.go:122] "RemoveContainer" containerID="829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d"
Oct 30 23:59:05.228256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.228221    4555 scope.go:122] "RemoveContainer" containerID="829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d"
Oct 30 23:59:05.228668 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:05.228639    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d\": container with ID starting with 829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d not found: ID does not exist" containerID="829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d"
Oct 30 23:59:05.228783 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.228683    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d"} err="failed to get container status \"829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d\": rpc error: code = NotFound desc = could not find container \"829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d\": container with ID starting with 829c795d600a54489791692e8cf4bd02074618d62d66712273dd34ac8eb6850d not found: ID does not exist"
Oct 30 23:59:05.244940 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.244885    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"]
Oct 30 23:59:05.249568 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.249494    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f0817c/test-model-f0817c-kserve-ff4d44fd9-gxz7v"]
Oct 30 23:59:05.252247 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.251836    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main" containerID="cri-o://f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847" gracePeriod=45
Oct 30 23:59:05.253336 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.253313    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648f4553-fa26-444b-a2e4-a690b559990f-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:05.253455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.253339    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brz44\" (UniqueName: \"kubernetes.io/projected/648f4553-fa26-444b-a2e4-a690b559990f-kube-api-access-brz44\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:05.253455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.253352    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:05.253455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.253382    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:05.253455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.253395    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:05.253455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.253406    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648f4553-fa26-444b-a2e4-a690b559990f-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:05.323091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.323052    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"]
Oct 30 23:59:05.323898 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.323558    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy" containerID="cri-o://e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38" gracePeriod=30
Oct 30 23:59:05.332341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.330769    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"]
Oct 30 23:59:05.629393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.628777    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.152:8000/ready\": dial tcp 10.133.0.152:8000: connect: connection refused"
Oct 30 23:59:05.699183 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.698823    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:05.844690 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.844656    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-openshift-ca\") pod \"f6680280-8130-4a71-a114-7d1a971ba1fb\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") "
Oct 30 23:59:05.844973 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.844951    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-xks-ca\") pod \"f6680280-8130-4a71-a114-7d1a971ba1fb\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") "
Oct 30 23:59:05.845032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.845007    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6680280-8130-4a71-a114-7d1a971ba1fb-kube-api-access-n4gxx\" (UniqueName: \"kubernetes.io/projected/f6680280-8130-4a71-a114-7d1a971ba1fb-kube-api-access-n4gxx\") pod \"f6680280-8130-4a71-a114-7d1a971ba1fb\" (UID: \"f6680280-8130-4a71-a114-7d1a971ba1fb\") "
Oct 30 23:59:05.845081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.845028    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-openshift-ca" pod "f6680280-8130-4a71-a114-7d1a971ba1fb" (UID: "f6680280-8130-4a71-a114-7d1a971ba1fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:05.845815 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.845789    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:05.847888 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.847860    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-xks-ca" pod "f6680280-8130-4a71-a114-7d1a971ba1fb" (UID: "f6680280-8130-4a71-a114-7d1a971ba1fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:05.847995 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.847954    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6680280-8130-4a71-a114-7d1a971ba1fb-kube-api-access-n4gxx" pod "f6680280-8130-4a71-a114-7d1a971ba1fb" (UID: "f6680280-8130-4a71-a114-7d1a971ba1fb"). InnerVolumeSpecName "kube-api-access-n4gxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:05.886834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.886762    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:59:05.953393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.951937    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6680280-8130-4a71-a114-7d1a971ba1fb-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:05.953393 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:05.951975    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4gxx\" (UniqueName: \"kubernetes.io/projected/f6680280-8130-4a71-a114-7d1a971ba1fb-kube-api-access-n4gxx\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:06.053354 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.053144    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-dshm\") pod \"462a2480-06db-45c5-8fc5-b0e5b714cadb\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") "
Oct 30 23:59:06.053354 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.053283    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-tmp-dir\") pod \"462a2480-06db-45c5-8fc5-b0e5b714cadb\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") "
Oct 30 23:59:06.053354 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.053330    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-model-cache\") pod \"462a2480-06db-45c5-8fc5-b0e5b714cadb\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") "
Oct 30 23:59:06.053354 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.053356    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462a2480-06db-45c5-8fc5-b0e5b714cadb-tls-certs\" (UniqueName: \"kubernetes.io/secret/462a2480-06db-45c5-8fc5-b0e5b714cadb-tls-certs\") pod \"462a2480-06db-45c5-8fc5-b0e5b714cadb\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") "
Oct 30 23:59:06.053788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.053397    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-home\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-home\") pod \"462a2480-06db-45c5-8fc5-b0e5b714cadb\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") "
Oct 30 23:59:06.053788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.053481    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462a2480-06db-45c5-8fc5-b0e5b714cadb-kube-api-access-zv2rl\" (UniqueName: \"kubernetes.io/projected/462a2480-06db-45c5-8fc5-b0e5b714cadb-kube-api-access-zv2rl\") pod \"462a2480-06db-45c5-8fc5-b0e5b714cadb\" (UID: \"462a2480-06db-45c5-8fc5-b0e5b714cadb\") "
Oct 30 23:59:06.053788 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.053675    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-model-cache" pod "462a2480-06db-45c5-8fc5-b0e5b714cadb" (UID: "462a2480-06db-45c5-8fc5-b0e5b714cadb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:06.054207 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.054156    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-home" pod "462a2480-06db-45c5-8fc5-b0e5b714cadb" (UID: "462a2480-06db-45c5-8fc5-b0e5b714cadb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:06.054826 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.054202    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-tmp-dir" pod "462a2480-06db-45c5-8fc5-b0e5b714cadb" (UID: "462a2480-06db-45c5-8fc5-b0e5b714cadb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:06.056284 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.056258    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:06.056284 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.056285    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:06.056630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.056297    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:06.056722 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.056627    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-dshm" pod "462a2480-06db-45c5-8fc5-b0e5b714cadb" (UID: "462a2480-06db-45c5-8fc5-b0e5b714cadb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:06.057566 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.057408    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462a2480-06db-45c5-8fc5-b0e5b714cadb-tls-certs" pod "462a2480-06db-45c5-8fc5-b0e5b714cadb" (UID: "462a2480-06db-45c5-8fc5-b0e5b714cadb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:06.057884 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.057862    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462a2480-06db-45c5-8fc5-b0e5b714cadb-kube-api-access-zv2rl" pod "462a2480-06db-45c5-8fc5-b0e5b714cadb" (UID: "462a2480-06db-45c5-8fc5-b0e5b714cadb"). InnerVolumeSpecName "kube-api-access-zv2rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:06.127203 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.127169    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"]
Oct 30 23:59:06.128003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.127982    4555 cpu_manager.go:414] "containerMap: removing container" podUID="29d54bb7-b0bc-4fd1-be96-9082e9dea41b" containerName="revoke-keys"
Oct 30 23:59:06.128094 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128004    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d54bb7-b0bc-4fd1-be96-9082e9dea41b" containerName="revoke-keys"
Oct 30 23:59:06.128094 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128022    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f6680280-8130-4a71-a114-7d1a971ba1fb" containerName="revoke-keys"
Oct 30 23:59:06.128094 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128029    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6680280-8130-4a71-a114-7d1a971ba1fb" containerName="revoke-keys"
Oct 30 23:59:06.128094 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128067    4555 cpu_manager.go:414] "containerMap: removing container" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main"
Oct 30 23:59:06.128094 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128073    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main"
Oct 30 23:59:06.128094 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128081    4555 cpu_manager.go:414] "containerMap: removing container" podUID="648f4553-fa26-444b-a2e4-a690b559990f" containerName="main"
Oct 30 23:59:06.128094 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128085    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648f4553-fa26-444b-a2e4-a690b559990f" containerName="main"
Oct 30 23:59:06.128523 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128188    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerName="main"
Oct 30 23:59:06.128523 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128197    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="648f4553-fa26-444b-a2e4-a690b559990f" containerName="main"
Oct 30 23:59:06.128523 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128206    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d54bb7-b0bc-4fd1-be96-9082e9dea41b" containerName="revoke-keys"
Oct 30 23:59:06.128523 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.128213    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6680280-8130-4a71-a114-7d1a971ba1fb" containerName="revoke-keys"
Oct 30 23:59:06.132116 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.132095    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.140347 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.140281    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"]
Oct 30 23:59:06.157478 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.157445    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462a2480-06db-45c5-8fc5-b0e5b714cadb-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:06.157817 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.157792    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv2rl\" (UniqueName: \"kubernetes.io/projected/462a2480-06db-45c5-8fc5-b0e5b714cadb-kube-api-access-zv2rl\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:06.157915 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.157825    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462a2480-06db-45c5-8fc5-b0e5b714cadb-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:06.224947 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.224912    4555 generic.go:356] "Generic (PLEG): container finished" podID="462a2480-06db-45c5-8fc5-b0e5b714cadb" containerID="f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847" exitCode=0
Oct 30 23:59:06.225539 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.224955    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"
Oct 30 23:59:06.225539 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.225037    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" event={"ID":"462a2480-06db-45c5-8fc5-b0e5b714cadb","Type":"ContainerDied","Data":"f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847"}
Oct 30 23:59:06.225539 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.225077    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4" event={"ID":"462a2480-06db-45c5-8fc5-b0e5b714cadb","Type":"ContainerDied","Data":"bd8af71f937ddc096b145af3fb485fe63d5a5806f0408b752ae11cb8096ce71d"}
Oct 30 23:59:06.225539 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.225103    4555 scope.go:122] "RemoveContainer" containerID="f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847"
Oct 30 23:59:06.226925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.226903    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"
Oct 30 23:59:06.227180 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.226924    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744" event={"ID":"f6680280-8130-4a71-a114-7d1a971ba1fb","Type":"ContainerDied","Data":"cee8c48312200f7917272b281c968421b59ff037fa05eed7558e6a343d3b7b46"}
Oct 30 23:59:06.227180 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.226971    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cee8c48312200f7917272b281c968421b59ff037fa05eed7558e6a343d3b7b46"
Oct 30 23:59:06.228553 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.228528    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy" containerID="cri-o://8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d" gracePeriod=30
Oct 30 23:59:06.239119 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.239093    4555 scope.go:122] "RemoveContainer" containerID="f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847"
Oct 30 23:59:06.240233 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:06.239388    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847\": container with ID starting with f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847 not found: ID does not exist" containerID="f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847"
Oct 30 23:59:06.240233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.239416    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847"} err="failed to get container status \"f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847\": rpc error: code = NotFound desc = could not find container \"f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847\": container with ID starting with f2832083b94bc602c514ceadc269fe316cb494b7e0903e5f7b692919a982e847 not found: ID does not exist"
Oct 30 23:59:06.258767 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.258734    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.258923 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.258853    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xnt\" (UniqueName: \"kubernetes.io/projected/63e88751-3dea-4c24-9bd0-ff016933c46f-kube-api-access-b9xnt\") pod \"maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.259007 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.258985    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.284046 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:06.284009    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod462a2480_06db_45c5_8fc5_b0e5b714cadb.slice/crio-bd8af71f937ddc096b145af3fb485fe63d5a5806f0408b752ae11cb8096ce71d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod462a2480_06db_45c5_8fc5_b0e5b714cadb.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 23:59:06.323359 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.323325    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"]
Oct 30 23:59:06.341792 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.341761    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7fc6e9/test-model-7fc6e9-kserve-6687bc68c5-brcr4"]
Oct 30 23:59:06.360486 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.360454    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.360635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.360539    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xnt\" (UniqueName: \"kubernetes.io/projected/63e88751-3dea-4c24-9bd0-ff016933c46f-kube-api-access-b9xnt\") pod \"maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.360635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.360611    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.361284 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.361264    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.363150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.363119    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.385519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.385487    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xnt\" (UniqueName: \"kubernetes.io/projected/63e88751-3dea-4c24-9bd0-ff016933c46f-kube-api-access-b9xnt\") pod \"maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.451721 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.451644    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:06.769193 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:59:06.769158    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e88751_3dea_4c24_9bd0_ff016933c46f.slice/crio-2949d1c70df325c6217853f0c3c0a87cf1939017f167ffb65fcc9615e63bb3ed WatchSource:0}: Error finding container 2949d1c70df325c6217853f0c3c0a87cf1939017f167ffb65fcc9615e63bb3ed: Status 404 returned error can't find the container with id 2949d1c70df325c6217853f0c3c0a87cf1939017f167ffb65fcc9615e63bb3ed
Oct 30 23:59:06.769325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:06.769262    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"]
Oct 30 23:59:07.210707 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:07.210670    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462a2480-06db-45c5-8fc5-b0e5b714cadb" path="/var/lib/kubelet/pods/462a2480-06db-45c5-8fc5-b0e5b714cadb/volumes"
Oct 30 23:59:07.211238 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:07.211223    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648f4553-fa26-444b-a2e4-a690b559990f" path="/var/lib/kubelet/pods/648f4553-fa26-444b-a2e4-a690b559990f/volumes"
Oct 30 23:59:07.238490 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:07.238442    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4" event={"ID":"63e88751-3dea-4c24-9bd0-ff016933c46f","Type":"ContainerStarted","Data":"5d799773ad23b9672450ec2d98787821b498985004d1e5e3bbf0a90efc18539f"}
Oct 30 23:59:07.238974 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:07.238511    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4" event={"ID":"63e88751-3dea-4c24-9bd0-ff016933c46f","Type":"ContainerStarted","Data":"2949d1c70df325c6217853f0c3c0a87cf1939017f167ffb65fcc9615e63bb3ed"}
Oct 30 23:59:07.264501 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:07.264434    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4" podStartSLOduration=1.2644156020000001 podStartE2EDuration="1.264415602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:07.260756929 +0000 UTC m=+2406.187394530" watchObservedRunningTime="2026-10-30 23:59:07.264415602 +0000 UTC m=+2406.191053183"
Oct 30 23:59:08.253229 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:08.253193    4555 generic.go:356] "Generic (PLEG): container finished" podID="63e88751-3dea-4c24-9bd0-ff016933c46f" containerID="5d799773ad23b9672450ec2d98787821b498985004d1e5e3bbf0a90efc18539f" exitCode=0
Oct 30 23:59:08.253785 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:08.253234    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4" event={"ID":"63e88751-3dea-4c24-9bd0-ff016933c46f","Type":"ContainerDied","Data":"5d799773ad23b9672450ec2d98787821b498985004d1e5e3bbf0a90efc18539f"}
Oct 30 23:59:09.785392 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:09.783868    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:09.848425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:09.845236    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-xks-ca\") pod \"63e88751-3dea-4c24-9bd0-ff016933c46f\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") "
Oct 30 23:59:09.848425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:09.845327    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-openshift-ca\") pod \"63e88751-3dea-4c24-9bd0-ff016933c46f\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") "
Oct 30 23:59:09.848425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:09.845659    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e88751-3dea-4c24-9bd0-ff016933c46f-kube-api-access-b9xnt\" (UniqueName: \"kubernetes.io/projected/63e88751-3dea-4c24-9bd0-ff016933c46f-kube-api-access-b9xnt\") pod \"63e88751-3dea-4c24-9bd0-ff016933c46f\" (UID: \"63e88751-3dea-4c24-9bd0-ff016933c46f\") "
Oct 30 23:59:09.848425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:09.846093    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-openshift-ca" pod "63e88751-3dea-4c24-9bd0-ff016933c46f" (UID: "63e88751-3dea-4c24-9bd0-ff016933c46f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:09.848425 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:09.846641    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:09.857417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:09.857384    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-xks-ca" pod "63e88751-3dea-4c24-9bd0-ff016933c46f" (UID: "63e88751-3dea-4c24-9bd0-ff016933c46f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:09.862398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:09.862346    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e88751-3dea-4c24-9bd0-ff016933c46f-kube-api-access-b9xnt" pod "63e88751-3dea-4c24-9bd0-ff016933c46f" (UID: "63e88751-3dea-4c24-9bd0-ff016933c46f"). InnerVolumeSpecName "kube-api-access-b9xnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:09.948165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:09.948116    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xnt\" (UniqueName: \"kubernetes.io/projected/63e88751-3dea-4c24-9bd0-ff016933c46f-kube-api-access-b9xnt\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:09.948165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:09.948150    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e88751-3dea-4c24-9bd0-ff016933c46f-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:10.362395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.362340    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4" event={"ID":"63e88751-3dea-4c24-9bd0-ff016933c46f","Type":"ContainerDied","Data":"2949d1c70df325c6217853f0c3c0a87cf1939017f167ffb65fcc9615e63bb3ed"}
Oct 30 23:59:10.362661 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.362635    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2949d1c70df325c6217853f0c3c0a87cf1939017f167ffb65fcc9615e63bb3ed"
Oct 30 23:59:10.362872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.362839    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"
Oct 30 23:59:10.642624 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.642594    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:59:10.680413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.680350    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-envoy\") pod \"0829125e-85cf-4a47-af4a-f5c820d7c513\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") "
Oct 30 23:59:10.681709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.681683    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-podinfo\") pod \"0829125e-85cf-4a47-af4a-f5c820d7c513\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") "
Oct 30 23:59:10.681901 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.681886    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-credential-socket\") pod \"0829125e-85cf-4a47-af4a-f5c820d7c513\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") "
Oct 30 23:59:10.682022 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.682009    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-certs\") pod \"0829125e-85cf-4a47-af4a-f5c820d7c513\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") "
Oct 30 23:59:10.682186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.682173    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-data\") pod \"0829125e-85cf-4a47-af4a-f5c820d7c513\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") "
Oct 30 23:59:10.682352 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.682334    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-kube-api-access-xxwjw\" (UniqueName: \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-kube-api-access-xxwjw\") pod \"0829125e-85cf-4a47-af4a-f5c820d7c513\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") "
Oct 30 23:59:10.682630 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.682609    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0829125e-85cf-4a47-af4a-f5c820d7c513-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0829125e-85cf-4a47-af4a-f5c820d7c513-istiod-ca-cert\") pod \"0829125e-85cf-4a47-af4a-f5c820d7c513\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") "
Oct 30 23:59:10.682836 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.682817    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-token\" (UniqueName: \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-token\") pod \"0829125e-85cf-4a47-af4a-f5c820d7c513\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") "
Oct 30 23:59:10.682986 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.682969    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-socket\") pod \"0829125e-85cf-4a47-af4a-f5c820d7c513\" (UID: \"0829125e-85cf-4a47-af4a-f5c820d7c513\") "
Oct 30 23:59:10.684061 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.684037    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-socket" pod "0829125e-85cf-4a47-af4a-f5c820d7c513" (UID: "0829125e-85cf-4a47-af4a-f5c820d7c513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:10.684869 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.684844    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-credential-socket" pod "0829125e-85cf-4a47-af4a-f5c820d7c513" (UID: "0829125e-85cf-4a47-af4a-f5c820d7c513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:10.685330 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.685308    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0829125e-85cf-4a47-af4a-f5c820d7c513-istiod-ca-cert" pod "0829125e-85cf-4a47-af4a-f5c820d7c513" (UID: "0829125e-85cf-4a47-af4a-f5c820d7c513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:10.689706 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.689678    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-certs" pod "0829125e-85cf-4a47-af4a-f5c820d7c513" (UID: "0829125e-85cf-4a47-af4a-f5c820d7c513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:10.691234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.691199    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-envoy" pod "0829125e-85cf-4a47-af4a-f5c820d7c513" (UID: "0829125e-85cf-4a47-af4a-f5c820d7c513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:10.691340 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.691248    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-podinfo" pod "0829125e-85cf-4a47-af4a-f5c820d7c513" (UID: "0829125e-85cf-4a47-af4a-f5c820d7c513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:59:10.691497 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.691473    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-token" pod "0829125e-85cf-4a47-af4a-f5c820d7c513" (UID: "0829125e-85cf-4a47-af4a-f5c820d7c513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:10.691574 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.691559    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-data" pod "0829125e-85cf-4a47-af4a-f5c820d7c513" (UID: "0829125e-85cf-4a47-af4a-f5c820d7c513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:10.705892 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.705859    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-kube-api-access-xxwjw" pod "0829125e-85cf-4a47-af4a-f5c820d7c513" (UID: "0829125e-85cf-4a47-af4a-f5c820d7c513"). InnerVolumeSpecName "kube-api-access-xxwjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:10.787582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.786053    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:10.787582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.786088    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:10.787582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.786104    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:10.787582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.786117    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:10.787582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.786130    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxwjw\" (UniqueName: \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-kube-api-access-xxwjw\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:10.787582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.786146    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0829125e-85cf-4a47-af4a-f5c820d7c513-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:10.787582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.786158    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:10.787582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.786171    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:10.787582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.786185    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0829125e-85cf-4a47-af4a-f5c820d7c513-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:10.924040 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.923949    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"]
Oct 30 23:59:10.925052 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.925029    4555 cpu_manager.go:414] "containerMap: removing container" podUID="63e88751-3dea-4c24-9bd0-ff016933c46f" containerName="revoke-keys"
Oct 30 23:59:10.925233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.925055    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e88751-3dea-4c24-9bd0-ff016933c46f" containerName="revoke-keys"
Oct 30 23:59:10.925233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.925076    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy"
Oct 30 23:59:10.925233 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.925085    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy"
Oct 30 23:59:10.925419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.925340    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e88751-3dea-4c24-9bd0-ff016933c46f" containerName="revoke-keys"
Oct 30 23:59:10.925419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.925356    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerName="istio-proxy"
Oct 30 23:59:10.933933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.933910    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:10.945695 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.945669    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e5934f\"/\"kube-root-ca.crt\""
Oct 30 23:59:10.945867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.945694    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e5934f\"/\"test-model-e5934f-kserve-self-signed-certs\""
Oct 30 23:59:10.945867 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.945774    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e5934f\"/\"default-dockercfg-wmzh5\""
Oct 30 23:59:10.950067 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.950034    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"]
Oct 30 23:59:10.994002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.993747    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-model-cache\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:10.994002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.993821    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tls-certs\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:10.994002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.993904    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tmp-dir\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:10.994324 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.994119    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-dshm\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:10.994324 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.994173    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcx7d\" (UniqueName: \"kubernetes.io/projected/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-kube-api-access-mcx7d\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:10.994324 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:10.994204    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-home\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.016863 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.016826    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"]
Oct 30 23:59:11.042434 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.042389    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e5934f\"/\"openshift-service-ca.crt\""
Oct 30 23:59:11.097791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.097204    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-dshm\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.097791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.097273    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcx7d\" (UniqueName: \"kubernetes.io/projected/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-kube-api-access-mcx7d\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.097791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.097304    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-home\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.097791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.097420    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-model-cache\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.097791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.097466    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tls-certs\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.097791 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.097537    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tmp-dir\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.098886 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.098577    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tmp-dir\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.099745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.099725    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-home\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.100768 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.100065    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-model-cache\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.108715 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.108634    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-dshm\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.111876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.111855    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tls-certs\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.116878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.116851    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcx7d\" (UniqueName: \"kubernetes.io/projected/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-kube-api-access-mcx7d\") pod \"test-model-e5934f-kserve-cf44c795-tv64v\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") " pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.255716 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.255673    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:11.394164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.394115    4555 generic.go:356] "Generic (PLEG): container finished" podID="0829125e-85cf-4a47-af4a-f5c820d7c513" containerID="e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38" exitCode=0
Oct 30 23:59:11.394455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.394413    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" event={"ID":"0829125e-85cf-4a47-af4a-f5c820d7c513","Type":"ContainerDied","Data":"e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38"}
Oct 30 23:59:11.394623 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.394599    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh" event={"ID":"0829125e-85cf-4a47-af4a-f5c820d7c513","Type":"ContainerDied","Data":"58fd5c290f0dd355e1b5ecacbd7d36c4948f16d8491bc2c2520f8cc080da6102"}
Oct 30 23:59:11.394748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.394735    4555 scope.go:122] "RemoveContainer" containerID="e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38"
Oct 30 23:59:11.395126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.395109    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"
Oct 30 23:59:11.425444 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.425076    4555 scope.go:122] "RemoveContainer" containerID="e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38"
Oct 30 23:59:11.428246 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:11.427361    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38\": container with ID starting with e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38 not found: ID does not exist" containerID="e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38"
Oct 30 23:59:11.428246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.427418    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38"} err="failed to get container status \"e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38\": rpc error: code = NotFound desc = could not find container \"e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38\": container with ID starting with e6d565d16324913562464bc99334a3608d283ae3d79a7e0d798c0a97cb16cf38 not found: ID does not exist"
Oct 30 23:59:11.428246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.428112    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"]
Oct 30 23:59:11.433133 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.433114    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-nshbh"]
Oct 30 23:59:11.676385 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.676342    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:59:11.716057 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.716027    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"]
Oct 30 23:59:11.732262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.731828    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-socket\") pod \"41036c28-a137-4175-9b83-5d368911b3ac\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") "
Oct 30 23:59:11.732262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.731912    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41036c28-a137-4175-9b83-5d368911b3ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41036c28-a137-4175-9b83-5d368911b3ac-istiod-ca-cert\") pod \"41036c28-a137-4175-9b83-5d368911b3ac\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") "
Oct 30 23:59:11.732262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.731944    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-credential-socket\") pod \"41036c28-a137-4175-9b83-5d368911b3ac\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") "
Oct 30 23:59:11.732262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.732021    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-certs\") pod \"41036c28-a137-4175-9b83-5d368911b3ac\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") "
Oct 30 23:59:11.732262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.732055    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-kube-api-access-jqv5d\" (UniqueName: \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-kube-api-access-jqv5d\") pod \"41036c28-a137-4175-9b83-5d368911b3ac\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") "
Oct 30 23:59:11.732262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.732112    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-istio-token\" (UniqueName: \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-istio-token\") pod \"41036c28-a137-4175-9b83-5d368911b3ac\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") "
Oct 30 23:59:11.732262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.732118    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-socket" pod "41036c28-a137-4175-9b83-5d368911b3ac" (UID: "41036c28-a137-4175-9b83-5d368911b3ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:11.732262 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.732226    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41036c28-a137-4175-9b83-5d368911b3ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41036c28-a137-4175-9b83-5d368911b3ac-istio-podinfo\") pod \"41036c28-a137-4175-9b83-5d368911b3ac\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") "
Oct 30 23:59:11.732804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.732319    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-data\") pod \"41036c28-a137-4175-9b83-5d368911b3ac\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") "
Oct 30 23:59:11.732804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.732459    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-envoy\") pod \"41036c28-a137-4175-9b83-5d368911b3ac\" (UID: \"41036c28-a137-4175-9b83-5d368911b3ac\") "
Oct 30 23:59:11.732804 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.732565    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41036c28-a137-4175-9b83-5d368911b3ac-istiod-ca-cert" pod "41036c28-a137-4175-9b83-5d368911b3ac" (UID: "41036c28-a137-4175-9b83-5d368911b3ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:11.733046 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.733028    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:11.733177 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.733160    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41036c28-a137-4175-9b83-5d368911b3ac-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:11.733237 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.733216    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-certs" pod "41036c28-a137-4175-9b83-5d368911b3ac" (UID: "41036c28-a137-4175-9b83-5d368911b3ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:11.734295 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.734048    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-credential-socket" pod "41036c28-a137-4175-9b83-5d368911b3ac" (UID: "41036c28-a137-4175-9b83-5d368911b3ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:11.734295 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.734268    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-data" pod "41036c28-a137-4175-9b83-5d368911b3ac" (UID: "41036c28-a137-4175-9b83-5d368911b3ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:11.735935 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.735912    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-kube-api-access-jqv5d" pod "41036c28-a137-4175-9b83-5d368911b3ac" (UID: "41036c28-a137-4175-9b83-5d368911b3ac"). InnerVolumeSpecName "kube-api-access-jqv5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:11.736059 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.736041    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-istio-token" pod "41036c28-a137-4175-9b83-5d368911b3ac" (UID: "41036c28-a137-4175-9b83-5d368911b3ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:11.736142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.736123    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-envoy" pod "41036c28-a137-4175-9b83-5d368911b3ac" (UID: "41036c28-a137-4175-9b83-5d368911b3ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:11.738315 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.738291    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41036c28-a137-4175-9b83-5d368911b3ac-istio-podinfo" pod "41036c28-a137-4175-9b83-5d368911b3ac" (UID: "41036c28-a137-4175-9b83-5d368911b3ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:59:11.834423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.834385    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:11.834423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.834417    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:11.834423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.834427    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqv5d\" (UniqueName: \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-kube-api-access-jqv5d\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:11.834835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.834436    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41036c28-a137-4175-9b83-5d368911b3ac-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:11.834835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.834445    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41036c28-a137-4175-9b83-5d368911b3ac-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:11.834835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.834454    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:11.834835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:11.834463    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41036c28-a137-4175-9b83-5d368911b3ac-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:12.301315 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.301280    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"]
Oct 30 23:59:12.309106 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.309076    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"]
Oct 30 23:59:12.408970 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.408926    4555 generic.go:356] "Generic (PLEG): container finished" podID="41036c28-a137-4175-9b83-5d368911b3ac" containerID="8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d" exitCode=0
Oct 30 23:59:12.409150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.408985    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"
Oct 30 23:59:12.409150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.409003    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" event={"ID":"41036c28-a137-4175-9b83-5d368911b3ac","Type":"ContainerDied","Data":"8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d"}
Oct 30 23:59:12.409150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.409061    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj" event={"ID":"41036c28-a137-4175-9b83-5d368911b3ac","Type":"ContainerDied","Data":"9af77714b52562101a577ffc5c6d6f96cdfb3c8750ca887782ecb36859630bd0"}
Oct 30 23:59:12.409150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.409091    4555 scope.go:122] "RemoveContainer" containerID="8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d"
Oct 30 23:59:12.411029 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.411007    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v" event={"ID":"153b8fe9-6fb0-4cdf-bf77-97b44eb644af","Type":"ContainerStarted","Data":"453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633"}
Oct 30 23:59:12.411142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.411043    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v" event={"ID":"153b8fe9-6fb0-4cdf-bf77-97b44eb644af","Type":"ContainerStarted","Data":"a9f667d91a785e5686466e1c577e8a4f10e979ac1dc7e2bcd4453b674c010ae6"}
Oct 30 23:59:12.423218 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.423196    4555 scope.go:122] "RemoveContainer" containerID="8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d"
Oct 30 23:59:12.423624 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:12.423603    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d\": container with ID starting with 8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d not found: ID does not exist" containerID="8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d"
Oct 30 23:59:12.423699 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.423631    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d"} err="failed to get container status \"8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d\": rpc error: code = NotFound desc = could not find container \"8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d\": container with ID starting with 8460c986638cd425e7a7bdf7633aee3c6fee088c21314cd2ff76a2f800d9941d not found: ID does not exist"
Oct 30 23:59:12.440014 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.439967    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v" podStartSLOduration=2.439953868 podStartE2EDuration="2.439953868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:12.435613965 +0000 UTC m=+2411.362251546" watchObservedRunningTime="2026-10-30 23:59:12.439953868 +0000 UTC m=+2411.366591448"
Oct 30 23:59:12.473471 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.473432    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"]
Oct 30 23:59:12.477961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:12.477935    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-23f75e-openshift-default-79bf5db5cc-m7kjj"]
Oct 30 23:59:13.213577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:13.213547    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0829125e-85cf-4a47-af4a-f5c820d7c513" path="/var/lib/kubelet/pods/0829125e-85cf-4a47-af4a-f5c820d7c513/volumes"
Oct 30 23:59:13.214341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:13.214311    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41036c28-a137-4175-9b83-5d368911b3ac" path="/var/lib/kubelet/pods/41036c28-a137-4175-9b83-5d368911b3ac/volumes"
Oct 30 23:59:13.258829 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:13.258801    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:13.445594 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:13.445564    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"]
Oct 30 23:59:13.446001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:13.445971    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy" containerID="cri-o://e23079989e9cb653203eef06c3a11a54a11b205148409368bd4a7a8b5a74f4c2" gracePeriod=30
Oct 30 23:59:13.453682 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:13.453657    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"]
Oct 30 23:59:13.454016 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:13.453991    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy" containerID="cri-o://3c135d038be4c16cba2ebf437fb8115bbc791e7d6bbe24872ad729168513ca30" gracePeriod=30
Oct 30 23:59:13.459069 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:13.459049    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"]
Oct 30 23:59:13.459272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:13.459252    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy" containerID="cri-o://5073d300c5d7359d181fcbb6151596c94e1f7e95cebf812c49ba23068b860944" gracePeriod=30
Oct 30 23:59:14.255648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.255613    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"]
Oct 30 23:59:14.256252 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.256238    4555 cpu_manager.go:414] "containerMap: removing container" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy"
Oct 30 23:59:14.256300 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.256253    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy"
Oct 30 23:59:14.256418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.256408    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="41036c28-a137-4175-9b83-5d368911b3ac" containerName="istio-proxy"
Oct 30 23:59:14.261845 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.261768    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:14.277416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.277356    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"]
Oct 30 23:59:14.309861 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.309827    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:59:14.310028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.309922    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:59:14.368709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.368670    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vm9f\" (UniqueName: \"kubernetes.io/projected/00ac5a69-e061-4847-98f5-8fae222fabb2-kube-api-access-9vm9f\") pod \"maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:14.368939 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.368827    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:14.369151 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.369128    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:14.471703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.470697    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:14.471703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.470812    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vm9f\" (UniqueName: \"kubernetes.io/projected/00ac5a69-e061-4847-98f5-8fae222fabb2-kube-api-access-9vm9f\") pod \"maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:14.471703 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.470859    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:14.471963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.471754    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:14.474794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.474762    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:14.492583 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.488969    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vm9f\" (UniqueName: \"kubernetes.io/projected/00ac5a69-e061-4847-98f5-8fae222fabb2-kube-api-access-9vm9f\") pod \"maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:14.546838 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.546802    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:59:14.547001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.546855    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:59:14.586458 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:14.586420    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:15.188074 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:59:15.188033    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ac5a69_e061_4847_98f5_8fae222fabb2.slice/crio-bd0a859c7611f11c8121f8b063398cb80ac367d8ca61540031f6c259ac75d6ec WatchSource:0}: Error finding container bd0a859c7611f11c8121f8b063398cb80ac367d8ca61540031f6c259ac75d6ec: Status 404 returned error can't find the container with id bd0a859c7611f11c8121f8b063398cb80ac367d8ca61540031f6c259ac75d6ec
Oct 30 23:59:15.191784 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.191444    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"]
Oct 30 23:59:15.391974 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.391905    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm" podUID="0cbd6143-2e09-46e2-b701-f1e457c32d1f" containerName="main" containerID="cri-o://a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40" gracePeriod=45
Oct 30 23:59:15.403495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.403465    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"]
Oct 30 23:59:15.410143 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.410117    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ccd42-cx744"]
Oct 30 23:59:15.541617 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.541581    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc" event={"ID":"00ac5a69-e061-4847-98f5-8fae222fabb2","Type":"ContainerStarted","Data":"bd0a859c7611f11c8121f8b063398cb80ac367d8ca61540031f6c259ac75d6ec"}
Oct 30 23:59:15.751719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.751656    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:59:15.840621 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.840589    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-home\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-home\") pod \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") "
Oct 30 23:59:15.840787 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.840707    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tls-certs\") pod \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") "
Oct 30 23:59:15.840842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.840821    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-dshm\") pod \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") "
Oct 30 23:59:15.840931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.840904    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-home" pod "0cbd6143-2e09-46e2-b701-f1e457c32d1f" (UID: "0cbd6143-2e09-46e2-b701-f1e457c32d1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:15.841009 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.840928    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tmp-dir\") pod \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") "
Oct 30 23:59:15.841075 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.841021    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-model-cache\") pod \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") "
Oct 30 23:59:15.841134 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.841083    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbd6143-2e09-46e2-b701-f1e457c32d1f-kube-api-access-gxw55\" (UniqueName: \"kubernetes.io/projected/0cbd6143-2e09-46e2-b701-f1e457c32d1f-kube-api-access-gxw55\") pod \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\" (UID: \"0cbd6143-2e09-46e2-b701-f1e457c32d1f\") "
Oct 30 23:59:15.841215 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.841193    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tmp-dir" pod "0cbd6143-2e09-46e2-b701-f1e457c32d1f" (UID: "0cbd6143-2e09-46e2-b701-f1e457c32d1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:15.841503 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.841299    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-model-cache" pod "0cbd6143-2e09-46e2-b701-f1e457c32d1f" (UID: "0cbd6143-2e09-46e2-b701-f1e457c32d1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:15.842677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.842648    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:15.842793 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.842687    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:15.842793 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.842706    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:15.843835 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.843808    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-dshm" pod "0cbd6143-2e09-46e2-b701-f1e457c32d1f" (UID: "0cbd6143-2e09-46e2-b701-f1e457c32d1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:15.844975 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.844938    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbd6143-2e09-46e2-b701-f1e457c32d1f-kube-api-access-gxw55" pod "0cbd6143-2e09-46e2-b701-f1e457c32d1f" (UID: "0cbd6143-2e09-46e2-b701-f1e457c32d1f"). InnerVolumeSpecName "kube-api-access-gxw55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:15.848540 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.848517    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tls-certs" pod "0cbd6143-2e09-46e2-b701-f1e457c32d1f" (UID: "0cbd6143-2e09-46e2-b701-f1e457c32d1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:15.943911 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.943880    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbd6143-2e09-46e2-b701-f1e457c32d1f-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:15.943911 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.943907    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbd6143-2e09-46e2-b701-f1e457c32d1f-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:15.943911 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:15.943917    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxw55\" (UniqueName: \"kubernetes.io/projected/0cbd6143-2e09-46e2-b701-f1e457c32d1f-kube-api-access-gxw55\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.026610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.026566    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerName="main" containerID="cri-o://4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad" gracePeriod=45
Oct 30 23:59:16.076314 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.076272    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l" podUID="68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" containerName="main" containerID="cri-o://d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1" gracePeriod=45
Oct 30 23:59:16.083053 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.083019    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l" podUID="68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": dial tcp 10.133.0.155:8000: connect: connection refused"
Oct 30 23:59:16.113318 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.113274    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": dial tcp 10.133.0.138:8000: connect: connection refused"
Oct 30 23:59:16.202462 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.202427    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:59:16.206530 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.202483    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:59:16.378296 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.377897    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:59:16.436209 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.436179    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:59:16.451816 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.451785    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-tmp-dir\") pod \"2490e605-01fd-4ce8-8911-611c95d43bd4\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") "
Oct 30 23:59:16.452001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.451845    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-home\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-home\") pod \"2490e605-01fd-4ce8-8911-611c95d43bd4\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") "
Oct 30 23:59:16.452001 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.451928    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-dshm\") pod \"2490e605-01fd-4ce8-8911-611c95d43bd4\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") "
Oct 30 23:59:16.452142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.452002    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-model-cache\") pod \"2490e605-01fd-4ce8-8911-611c95d43bd4\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") "
Oct 30 23:59:16.452142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.452048    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2490e605-01fd-4ce8-8911-611c95d43bd4-kube-api-access-6whmc\" (UniqueName: \"kubernetes.io/projected/2490e605-01fd-4ce8-8911-611c95d43bd4-kube-api-access-6whmc\") pod \"2490e605-01fd-4ce8-8911-611c95d43bd4\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") "
Oct 30 23:59:16.452142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.452078    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2490e605-01fd-4ce8-8911-611c95d43bd4-tls-certs\" (UniqueName: \"kubernetes.io/secret/2490e605-01fd-4ce8-8911-611c95d43bd4-tls-certs\") pod \"2490e605-01fd-4ce8-8911-611c95d43bd4\" (UID: \"2490e605-01fd-4ce8-8911-611c95d43bd4\") "
Oct 30 23:59:16.452142 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.452073    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-tmp-dir" pod "2490e605-01fd-4ce8-8911-611c95d43bd4" (UID: "2490e605-01fd-4ce8-8911-611c95d43bd4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:16.452798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.452772    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-model-cache" pod "2490e605-01fd-4ce8-8911-611c95d43bd4" (UID: "2490e605-01fd-4ce8-8911-611c95d43bd4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:16.453093 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.452819    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-home" pod "2490e605-01fd-4ce8-8911-611c95d43bd4" (UID: "2490e605-01fd-4ce8-8911-611c95d43bd4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:16.453093 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.452836    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.455503 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.455352    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2490e605-01fd-4ce8-8911-611c95d43bd4-kube-api-access-6whmc" pod "2490e605-01fd-4ce8-8911-611c95d43bd4" (UID: "2490e605-01fd-4ce8-8911-611c95d43bd4"). InnerVolumeSpecName "kube-api-access-6whmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:16.455723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.455695    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-dshm" pod "2490e605-01fd-4ce8-8911-611c95d43bd4" (UID: "2490e605-01fd-4ce8-8911-611c95d43bd4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:16.457321 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.457293    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2490e605-01fd-4ce8-8911-611c95d43bd4-tls-certs" pod "2490e605-01fd-4ce8-8911-611c95d43bd4" (UID: "2490e605-01fd-4ce8-8911-611c95d43bd4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:16.457984 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.457870    4555 patch_prober.go:29] interesting pod/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:59:16.457984 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.457929    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:59:16.552656 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.552621    4555 generic.go:356] "Generic (PLEG): container finished" podID="68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" containerID="d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1" exitCode=0
Oct 30 23:59:16.552848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.552669    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"
Oct 30 23:59:16.552848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.552707    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l" event={"ID":"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce","Type":"ContainerDied","Data":"d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1"}
Oct 30 23:59:16.552848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.552755    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l" event={"ID":"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce","Type":"ContainerDied","Data":"753cfe5f49df6f00174410082c0643880f9b7960ec5d4f2bb081483963fcd683"}
Oct 30 23:59:16.552848 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.552791    4555 scope.go:122] "RemoveContainer" containerID="d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1"
Oct 30 23:59:16.553385 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.553305    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-dshm\") pod \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") "
Oct 30 23:59:16.553509 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.553417    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-kube-api-access-bg5mb\" (UniqueName: \"kubernetes.io/projected/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-kube-api-access-bg5mb\") pod \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") "
Oct 30 23:59:16.553560 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.553523    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tmp-dir\") pod \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") "
Oct 30 23:59:16.553560 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.553554    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-model-cache\") pod \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") "
Oct 30 23:59:16.553668 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.553576    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tls-certs\") pod \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") "
Oct 30 23:59:16.553668 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.553640    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-home\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-home\") pod \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\" (UID: \"68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce\") "
Oct 30 23:59:16.553852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.553828    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tmp-dir" pod "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" (UID: "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:16.554000 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.553962    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-model-cache" pod "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" (UID: "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:16.554061 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.554026    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-home" pod "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" (UID: "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:16.554334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.554316    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.554334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.554339    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.554587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.554352    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.554587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.554408    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.554587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.554426    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2490e605-01fd-4ce8-8911-611c95d43bd4-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.554587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.554440    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.554587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.554456    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6whmc\" (UniqueName: \"kubernetes.io/projected/2490e605-01fd-4ce8-8911-611c95d43bd4-kube-api-access-6whmc\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.554587 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.554470    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2490e605-01fd-4ce8-8911-611c95d43bd4-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.555569 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.555543    4555 generic.go:356] "Generic (PLEG): container finished" podID="00ac5a69-e061-4847-98f5-8fae222fabb2" containerID="7ff5981afb67083bc171c02fd2eca9943c4bacafcb12457437e93292fac5a09d" exitCode=0
Oct 30 23:59:16.555723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.555701    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc" event={"ID":"00ac5a69-e061-4847-98f5-8fae222fabb2","Type":"ContainerDied","Data":"7ff5981afb67083bc171c02fd2eca9943c4bacafcb12457437e93292fac5a09d"}
Oct 30 23:59:16.556591 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.556562    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tls-certs" pod "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" (UID: "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:16.557019 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.556992    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-dshm" pod "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" (UID: "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:16.557128 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.556993    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-kube-api-access-bg5mb" pod "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" (UID: "68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce"). InnerVolumeSpecName "kube-api-access-bg5mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:16.557714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.557693    4555 generic.go:356] "Generic (PLEG): container finished" podID="0cbd6143-2e09-46e2-b701-f1e457c32d1f" containerID="a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40" exitCode=0
Oct 30 23:59:16.557872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.557749    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm" event={"ID":"0cbd6143-2e09-46e2-b701-f1e457c32d1f","Type":"ContainerDied","Data":"a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40"}
Oct 30 23:59:16.557872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.557787    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm" event={"ID":"0cbd6143-2e09-46e2-b701-f1e457c32d1f","Type":"ContainerDied","Data":"38bacb6e48b2a475b40c96d5da9645f012898fa82ec2fdf6fabb486dc9a4e695"}
Oct 30 23:59:16.557872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.557757    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"
Oct 30 23:59:16.559515 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.559488    4555 generic.go:356] "Generic (PLEG): container finished" podID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerID="4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad" exitCode=0
Oct 30 23:59:16.559629 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.559518    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"
Oct 30 23:59:16.559629 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.559545    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" event={"ID":"2490e605-01fd-4ce8-8911-611c95d43bd4","Type":"ContainerDied","Data":"4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad"}
Oct 30 23:59:16.559629 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.559571    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64" event={"ID":"2490e605-01fd-4ce8-8911-611c95d43bd4","Type":"ContainerDied","Data":"4a3708ceebecba4f9b47782337fbff85f24f8ef3ed621bb0356d7b4b26ba4f32"}
Oct 30 23:59:16.590040 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.590011    4555 scope.go:122] "RemoveContainer" containerID="d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1"
Oct 30 23:59:16.590544 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:16.590512    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1\": container with ID starting with d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1 not found: ID does not exist" containerID="d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1"
Oct 30 23:59:16.590665 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.590555    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1"} err="failed to get container status \"d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1\": rpc error: code = NotFound desc = could not find container \"d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1\": container with ID starting with d37ef8f42586fe8e9957acba4964cc9a8cb2ec6152881f0a750c4230bf20d2a1 not found: ID does not exist"
Oct 30 23:59:16.590665 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.590615    4555 scope.go:122] "RemoveContainer" containerID="a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40"
Oct 30 23:59:16.603617 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.603591    4555 scope.go:122] "RemoveContainer" containerID="a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40"
Oct 30 23:59:16.604092 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:16.604066    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40\": container with ID starting with a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40 not found: ID does not exist" containerID="a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40"
Oct 30 23:59:16.604186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.604100    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40"} err="failed to get container status \"a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40\": rpc error: code = NotFound desc = could not find container \"a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40\": container with ID starting with a0ad3d598ad883deeabbb1e9eed01a7aaaa35d5826a0740ffcded2ad2b0e9c40 not found: ID does not exist"
Oct 30 23:59:16.604186 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.604117    4555 scope.go:122] "RemoveContainer" containerID="4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad"
Oct 30 23:59:16.615042 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.615009    4555 scope.go:122] "RemoveContainer" containerID="4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad"
Oct 30 23:59:16.615299 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:16.615278    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad\": container with ID starting with 4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad not found: ID does not exist" containerID="4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad"
Oct 30 23:59:16.615405 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.615308    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad"} err="failed to get container status \"4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad\": rpc error: code = NotFound desc = could not find container \"4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad\": container with ID starting with 4ab18ac0bf4db2daecf412385930427e1ba6615478c0652f8320ff3de67b52ad not found: ID does not exist"
Oct 30 23:59:16.619530 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.619508    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"]
Oct 30 23:59:16.638278 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.638250    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2ccd42/test-model-2ccd42-kserve-6856ff8f9f-xgl64"]
Oct 30 23:59:16.655771 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.655700    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.655931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.655795    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.655931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.655814    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg5mb\" (UniqueName: \"kubernetes.io/projected/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce-kube-api-access-bg5mb\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:16.659048 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.659022    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"]
Oct 30 23:59:16.663564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.663536    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a63288/test-model-a63288-kserve-5b8bfccbf5-pdxhm"]
Oct 30 23:59:16.956033 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.956000    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"]
Oct 30 23:59:16.961410 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:16.961359    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-32732d/test-model-32732d-kserve-5ff7dbc97-6cp6l"]
Oct 30 23:59:17.005360 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.005322    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerName="main" containerID="cri-o://ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee" gracePeriod=45
Oct 30 23:59:17.143143 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.143068    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": dial tcp 10.133.0.142:8000: connect: connection refused"
Oct 30 23:59:17.284847 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.272797    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbd6143-2e09-46e2-b701-f1e457c32d1f" path="/var/lib/kubelet/pods/0cbd6143-2e09-46e2-b701-f1e457c32d1f/volumes"
Oct 30 23:59:17.299985 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.299947    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" path="/var/lib/kubelet/pods/2490e605-01fd-4ce8-8911-611c95d43bd4/volumes"
Oct 30 23:59:17.300659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.300636    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" path="/var/lib/kubelet/pods/68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce/volumes"
Oct 30 23:59:17.301429 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.301407    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6680280-8130-4a71-a114-7d1a971ba1fb" path="/var/lib/kubelet/pods/f6680280-8130-4a71-a114-7d1a971ba1fb/volumes"
Oct 30 23:59:17.301915 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.301891    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm" podUID="e76a92f7-ee45-4e29-bac6-2d47c6557755" containerName="payload-processing" containerID="cri-o://0ba0902feca7af97586366411a63507cf3a77e8de8c872d7a592119c32ddda59" gracePeriod=25
Oct 30 23:59:17.310251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.310085    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv" podUID="85fac7c9-babe-422f-96f4-d58352606a54" containerName="payload-pre-processing" containerID="cri-o://f1b39fa07183fc2e80e89e844e2654efd42a331e269085e05a95726dded168cb" gracePeriod=25
Oct 30 23:59:17.451656 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.451627    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:59:17.489331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.488206    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b014885-3f0e-43d6-bf70-de300c1ae2db-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b014885-3f0e-43d6-bf70-de300c1ae2db-tls-certs\") pod \"1b014885-3f0e-43d6-bf70-de300c1ae2db\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") "
Oct 30 23:59:17.489331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.488277    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-dshm\") pod \"1b014885-3f0e-43d6-bf70-de300c1ae2db\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") "
Oct 30 23:59:17.489331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.488436    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b014885-3f0e-43d6-bf70-de300c1ae2db-kube-api-access-gcs2d\" (UniqueName: \"kubernetes.io/projected/1b014885-3f0e-43d6-bf70-de300c1ae2db-kube-api-access-gcs2d\") pod \"1b014885-3f0e-43d6-bf70-de300c1ae2db\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") "
Oct 30 23:59:17.489331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.488487    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-home\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-home\") pod \"1b014885-3f0e-43d6-bf70-de300c1ae2db\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") "
Oct 30 23:59:17.489331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.488519    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-model-cache\") pod \"1b014885-3f0e-43d6-bf70-de300c1ae2db\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") "
Oct 30 23:59:17.489331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.488657    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-tmp-dir\") pod \"1b014885-3f0e-43d6-bf70-de300c1ae2db\" (UID: \"1b014885-3f0e-43d6-bf70-de300c1ae2db\") "
Oct 30 23:59:17.489331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.489050    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-model-cache" pod "1b014885-3f0e-43d6-bf70-de300c1ae2db" (UID: "1b014885-3f0e-43d6-bf70-de300c1ae2db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:17.489331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.489138    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-home" pod "1b014885-3f0e-43d6-bf70-de300c1ae2db" (UID: "1b014885-3f0e-43d6-bf70-de300c1ae2db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:17.489331 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.489154    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-tmp-dir" pod "1b014885-3f0e-43d6-bf70-de300c1ae2db" (UID: "1b014885-3f0e-43d6-bf70-de300c1ae2db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:17.490253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.489481    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:17.490253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.489500    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:17.490253 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.489514    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:17.508486 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.492476    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b014885-3f0e-43d6-bf70-de300c1ae2db-tls-certs" pod "1b014885-3f0e-43d6-bf70-de300c1ae2db" (UID: "1b014885-3f0e-43d6-bf70-de300c1ae2db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:17.508486 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.492510    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b014885-3f0e-43d6-bf70-de300c1ae2db-kube-api-access-gcs2d" pod "1b014885-3f0e-43d6-bf70-de300c1ae2db" (UID: "1b014885-3f0e-43d6-bf70-de300c1ae2db"). InnerVolumeSpecName "kube-api-access-gcs2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:17.520651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.518311    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-dshm" pod "1b014885-3f0e-43d6-bf70-de300c1ae2db" (UID: "1b014885-3f0e-43d6-bf70-de300c1ae2db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:17.586060 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.585475    4555 generic.go:356] "Generic (PLEG): container finished" podID="85fac7c9-babe-422f-96f4-d58352606a54" containerID="f1b39fa07183fc2e80e89e844e2654efd42a331e269085e05a95726dded168cb" exitCode=0
Oct 30 23:59:17.586060 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.585585    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv" event={"ID":"85fac7c9-babe-422f-96f4-d58352606a54","Type":"ContainerDied","Data":"f1b39fa07183fc2e80e89e844e2654efd42a331e269085e05a95726dded168cb"}
Oct 30 23:59:17.588503 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.588006    4555 generic.go:356] "Generic (PLEG): container finished" podID="e76a92f7-ee45-4e29-bac6-2d47c6557755" containerID="0ba0902feca7af97586366411a63507cf3a77e8de8c872d7a592119c32ddda59" exitCode=0
Oct 30 23:59:17.588503 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.588081    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm" event={"ID":"e76a92f7-ee45-4e29-bac6-2d47c6557755","Type":"ContainerDied","Data":"0ba0902feca7af97586366411a63507cf3a77e8de8c872d7a592119c32ddda59"}
Oct 30 23:59:17.591964 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.591937    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b014885-3f0e-43d6-bf70-de300c1ae2db-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:17.592069 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.591967    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b014885-3f0e-43d6-bf70-de300c1ae2db-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:17.592069 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.591982    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcs2d\" (UniqueName: \"kubernetes.io/projected/1b014885-3f0e-43d6-bf70-de300c1ae2db-kube-api-access-gcs2d\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:17.596979 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.596939    4555 generic.go:356] "Generic (PLEG): container finished" podID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerID="ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee" exitCode=0
Oct 30 23:59:17.612963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.597127    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" event={"ID":"1b014885-3f0e-43d6-bf70-de300c1ae2db","Type":"ContainerDied","Data":"ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee"}
Oct 30 23:59:17.612963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.597162    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g" event={"ID":"1b014885-3f0e-43d6-bf70-de300c1ae2db","Type":"ContainerDied","Data":"655b49fffa2587b570383e3c16d87fc30739ea2d3804a8973c74c1b3fde3e47a"}
Oct 30 23:59:17.612963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.597184    4555 scope.go:122] "RemoveContainer" containerID="ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee"
Oct 30 23:59:17.612963 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.597296    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"
Oct 30 23:59:17.659408 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.659306    4555 scope.go:122] "RemoveContainer" containerID="ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee"
Oct 30 23:59:17.660225 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:17.660130    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee\": container with ID starting with ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee not found: ID does not exist" containerID="ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee"
Oct 30 23:59:17.660225 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.660168    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee"} err="failed to get container status \"ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee\": rpc error: code = NotFound desc = could not find container \"ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee\": container with ID starting with ed33a52e84183ac578cbf56375c1b197bc60f0f2ea35edc7133b2899b86bb7ee not found: ID does not exist"
Oct 30 23:59:17.673081 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.673024    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"]
Oct 30 23:59:17.673564 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.673527    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:59:17.678208 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.677503    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:59:17.679663 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.679644    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1bc837/test-model-1bc837-kserve-744b8cb76c-kq57g"]
Oct 30 23:59:17.828445 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.828408    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76a92f7-ee45-4e29-bac6-2d47c6557755-kube-api-access-j4gpv\" (UniqueName: \"kubernetes.io/projected/e76a92f7-ee45-4e29-bac6-2d47c6557755-kube-api-access-j4gpv\") pod \"e76a92f7-ee45-4e29-bac6-2d47c6557755\" (UID: \"e76a92f7-ee45-4e29-bac6-2d47c6557755\") "
Oct 30 23:59:17.828635 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.828621    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85fac7c9-babe-422f-96f4-d58352606a54-kube-api-access-wsbb7\" (UniqueName: \"kubernetes.io/projected/85fac7c9-babe-422f-96f4-d58352606a54-kube-api-access-wsbb7\") pod \"85fac7c9-babe-422f-96f4-d58352606a54\" (UID: \"85fac7c9-babe-422f-96f4-d58352606a54\") "
Oct 30 23:59:17.828770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.828744    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85fac7c9-babe-422f-96f4-d58352606a54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fac7c9-babe-422f-96f4-d58352606a54-plugins-config-volume\") pod \"85fac7c9-babe-422f-96f4-d58352606a54\" (UID: \"85fac7c9-babe-422f-96f4-d58352606a54\") "
Oct 30 23:59:17.828834 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.828788    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76a92f7-ee45-4e29-bac6-2d47c6557755-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76a92f7-ee45-4e29-bac6-2d47c6557755-plugins-config-volume\") pod \"e76a92f7-ee45-4e29-bac6-2d47c6557755\" (UID: \"e76a92f7-ee45-4e29-bac6-2d47c6557755\") "
Oct 30 23:59:17.829328 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.829297    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85fac7c9-babe-422f-96f4-d58352606a54-plugins-config-volume" pod "85fac7c9-babe-422f-96f4-d58352606a54" (UID: "85fac7c9-babe-422f-96f4-d58352606a54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:17.829456 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.829426    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76a92f7-ee45-4e29-bac6-2d47c6557755-plugins-config-volume" pod "e76a92f7-ee45-4e29-bac6-2d47c6557755" (UID: "e76a92f7-ee45-4e29-bac6-2d47c6557755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:17.829672 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.829646    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fac7c9-babe-422f-96f4-d58352606a54-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:17.829672 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.829668    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76a92f7-ee45-4e29-bac6-2d47c6557755-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:17.838787 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.838756    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76a92f7-ee45-4e29-bac6-2d47c6557755-kube-api-access-j4gpv" pod "e76a92f7-ee45-4e29-bac6-2d47c6557755" (UID: "e76a92f7-ee45-4e29-bac6-2d47c6557755"). InnerVolumeSpecName "kube-api-access-j4gpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:17.842080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.841913    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85fac7c9-babe-422f-96f4-d58352606a54-kube-api-access-wsbb7" pod "85fac7c9-babe-422f-96f4-d58352606a54" (UID: "85fac7c9-babe-422f-96f4-d58352606a54"). InnerVolumeSpecName "kube-api-access-wsbb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:17.881157 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.881129    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:17.933165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.933063    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gpv\" (UniqueName: \"kubernetes.io/projected/e76a92f7-ee45-4e29-bac6-2d47c6557755-kube-api-access-j4gpv\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:17.933165 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:17.933104    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsbb7\" (UniqueName: \"kubernetes.io/projected/85fac7c9-babe-422f-96f4-d58352606a54-kube-api-access-wsbb7\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:18.034602 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.034566    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-openshift-ca\") pod \"00ac5a69-e061-4847-98f5-8fae222fabb2\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") "
Oct 30 23:59:18.034798 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.034673    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ac5a69-e061-4847-98f5-8fae222fabb2-kube-api-access-9vm9f\" (UniqueName: \"kubernetes.io/projected/00ac5a69-e061-4847-98f5-8fae222fabb2-kube-api-access-9vm9f\") pod \"00ac5a69-e061-4847-98f5-8fae222fabb2\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") "
Oct 30 23:59:18.034878 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.034847    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-xks-ca\") pod \"00ac5a69-e061-4847-98f5-8fae222fabb2\" (UID: \"00ac5a69-e061-4847-98f5-8fae222fabb2\") "
Oct 30 23:59:18.034993 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.034968    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-openshift-ca" pod "00ac5a69-e061-4847-98f5-8fae222fabb2" (UID: "00ac5a69-e061-4847-98f5-8fae222fabb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:18.035707 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.035683    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:18.037646 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.037614    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-xks-ca" pod "00ac5a69-e061-4847-98f5-8fae222fabb2" (UID: "00ac5a69-e061-4847-98f5-8fae222fabb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:18.037742 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.037647    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ac5a69-e061-4847-98f5-8fae222fabb2-kube-api-access-9vm9f" pod "00ac5a69-e061-4847-98f5-8fae222fabb2" (UID: "00ac5a69-e061-4847-98f5-8fae222fabb2"). InnerVolumeSpecName "kube-api-access-9vm9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:18.147397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.146891    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ac5a69-e061-4847-98f5-8fae222fabb2-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:18.147397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.146932    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vm9f\" (UniqueName: \"kubernetes.io/projected/00ac5a69-e061-4847-98f5-8fae222fabb2-kube-api-access-9vm9f\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:18.634309 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.634279    4555 generic.go:356] "Generic (PLEG): container finished" podID="0016f269-b213-41a9-b78e-5725aac9de1e" containerID="e23079989e9cb653203eef06c3a11a54a11b205148409368bd4a7a8b5a74f4c2" exitCode=0
Oct 30 23:59:18.634818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.634567    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" event={"ID":"0016f269-b213-41a9-b78e-5725aac9de1e","Type":"ContainerDied","Data":"e23079989e9cb653203eef06c3a11a54a11b205148409368bd4a7a8b5a74f4c2"}
Oct 30 23:59:18.634818 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.634643    4555 scope.go:122] "RemoveContainer" containerID="8d54a0292141e6f0cf7d6ab2628ca5d7091a02cbb1baf23f2811be200a5d61e7"
Oct 30 23:59:18.637173 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.637099    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv" event={"ID":"85fac7c9-babe-422f-96f4-d58352606a54","Type":"ContainerDied","Data":"ac20f694bb7b0692f16bc6c76ea806253390bf4dfc9f01c2a1904636710a0856"}
Oct 30 23:59:18.637317 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.637248    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"
Oct 30 23:59:18.642769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.642573    4555 generic.go:356] "Generic (PLEG): container finished" podID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerID="5073d300c5d7359d181fcbb6151596c94e1f7e95cebf812c49ba23068b860944" exitCode=0
Oct 30 23:59:18.642769 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.642712    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" event={"ID":"9d4c30d5-2321-45c8-8a25-ce66a86629a6","Type":"ContainerDied","Data":"5073d300c5d7359d181fcbb6151596c94e1f7e95cebf812c49ba23068b860944"}
Oct 30 23:59:18.645353 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.645267    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm" event={"ID":"e76a92f7-ee45-4e29-bac6-2d47c6557755","Type":"ContainerDied","Data":"f0541a00ddf1920fb256056c670e9018d646de036eb996e690bc4d8c205d5f90"}
Oct 30 23:59:18.645727 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.645679    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"
Oct 30 23:59:18.652296 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.652220    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc" event={"ID":"00ac5a69-e061-4847-98f5-8fae222fabb2","Type":"ContainerDied","Data":"bd0a859c7611f11c8121f8b063398cb80ac367d8ca61540031f6c259ac75d6ec"}
Oct 30 23:59:18.652296 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.652262    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd0a859c7611f11c8121f8b063398cb80ac367d8ca61540031f6c259ac75d6ec"
Oct 30 23:59:18.652475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.652362    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"
Oct 30 23:59:18.671561 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.667644    4555 generic.go:356] "Generic (PLEG): container finished" podID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerID="3c135d038be4c16cba2ebf437fb8115bbc791e7d6bbe24872ad729168513ca30" exitCode=0
Oct 30 23:59:18.671561 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.667763    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" event={"ID":"77e0b9a5-206c-4589-a9dc-4b4f46620605","Type":"ContainerDied","Data":"3c135d038be4c16cba2ebf437fb8115bbc791e7d6bbe24872ad729168513ca30"}
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.678093    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"]
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679093    4555 cpu_manager.go:414] "containerMap: removing container" podUID="e76a92f7-ee45-4e29-bac6-2d47c6557755" containerName="payload-processing"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679115    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76a92f7-ee45-4e29-bac6-2d47c6557755" containerName="payload-processing"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679130    4555 cpu_manager.go:414] "containerMap: removing container" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerName="main"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679138    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerName="main"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679154    4555 cpu_manager.go:414] "containerMap: removing container" podUID="68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" containerName="main"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679163    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" containerName="main"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679184    4555 cpu_manager.go:414] "containerMap: removing container" podUID="00ac5a69-e061-4847-98f5-8fae222fabb2" containerName="revoke-keys"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679195    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ac5a69-e061-4847-98f5-8fae222fabb2" containerName="revoke-keys"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679227    4555 cpu_manager.go:414] "containerMap: removing container" podUID="85fac7c9-babe-422f-96f4-d58352606a54" containerName="payload-pre-processing"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679239    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85fac7c9-babe-422f-96f4-d58352606a54" containerName="payload-pre-processing"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679279    4555 cpu_manager.go:414] "containerMap: removing container" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerName="main"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679286    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerName="main"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679300    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0cbd6143-2e09-46e2-b701-f1e457c32d1f" containerName="main"
Oct 30 23:59:18.679357 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679307    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbd6143-2e09-46e2-b701-f1e457c32d1f" containerName="main"
Oct 30 23:59:18.680356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679485    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76a92f7-ee45-4e29-bac6-2d47c6557755" containerName="payload-processing"
Oct 30 23:59:18.680356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679504    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="2490e605-01fd-4ce8-8911-611c95d43bd4" containerName="main"
Oct 30 23:59:18.680356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679517    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f95bc2-a77c-4e6d-aeb2-48ac03d9e8ce" containerName="main"
Oct 30 23:59:18.680356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679529    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ac5a69-e061-4847-98f5-8fae222fabb2" containerName="revoke-keys"
Oct 30 23:59:18.680356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679542    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="85fac7c9-babe-422f-96f4-d58352606a54" containerName="payload-pre-processing"
Oct 30 23:59:18.680356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679558    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" containerName="main"
Oct 30 23:59:18.680356 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.679565    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbd6143-2e09-46e2-b701-f1e457c32d1f" containerName="main"
Oct 30 23:59:18.691755 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.685977    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:18.714511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.708173    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:59:18.714511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.708228    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:59:18.719811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.719452    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"]
Oct 30 23:59:18.734942 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.730924    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"]
Oct 30 23:59:18.738531 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.738500    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ccd42-dcbb5c7f5-mv6zv"]
Oct 30 23:59:18.740064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.740035    4555 scope.go:122] "RemoveContainer" containerID="f1b39fa07183fc2e80e89e844e2654efd42a331e269085e05a95726dded168cb"
Oct 30 23:59:18.759648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.759607    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m664b\" (UniqueName: \"kubernetes.io/projected/265af9b2-d3a0-465a-8200-7db6c861b0ad-kube-api-access-m664b\") pod \"maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:18.759895 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.759871    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:18.760032 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.760012    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:18.761714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.761690    4555 scope.go:122] "RemoveContainer" containerID="0ba0902feca7af97586366411a63507cf3a77e8de8c872d7a592119c32ddda59"
Oct 30 23:59:18.798277 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.798225    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"]
Oct 30 23:59:18.804972 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.804938    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ccd42-74fd479f69-kgkbm"]
Oct 30 23:59:18.831241 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.830944    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:59:18.923041 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.923005    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-certs\") pod \"0016f269-b213-41a9-b78e-5725aac9de1e\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") "
Oct 30 23:59:18.923882 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.923857    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-data\") pod \"0016f269-b213-41a9-b78e-5725aac9de1e\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") "
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.923771    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-certs" pod "0016f269-b213-41a9-b78e-5725aac9de1e" (UID: "0016f269-b213-41a9-b78e-5725aac9de1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.924164    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-credential-socket\") pod \"0016f269-b213-41a9-b78e-5725aac9de1e\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") "
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.924313    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-socket\") pod \"0016f269-b213-41a9-b78e-5725aac9de1e\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") "
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.924427    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-kube-api-access-g6lwn\" (UniqueName: \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-kube-api-access-g6lwn\") pod \"0016f269-b213-41a9-b78e-5725aac9de1e\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") "
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.924431    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-credential-socket" pod "0016f269-b213-41a9-b78e-5725aac9de1e" (UID: "0016f269-b213-41a9-b78e-5725aac9de1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.924461    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-envoy\") pod \"0016f269-b213-41a9-b78e-5725aac9de1e\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") "
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.924505    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0016f269-b213-41a9-b78e-5725aac9de1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0016f269-b213-41a9-b78e-5725aac9de1e-istiod-ca-cert\") pod \"0016f269-b213-41a9-b78e-5725aac9de1e\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") "
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.924557    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-istio-token\" (UniqueName: \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-istio-token\") pod \"0016f269-b213-41a9-b78e-5725aac9de1e\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") "
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.924586    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0016f269-b213-41a9-b78e-5725aac9de1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0016f269-b213-41a9-b78e-5725aac9de1e-istio-podinfo\") pod \"0016f269-b213-41a9-b78e-5725aac9de1e\" (UID: \"0016f269-b213-41a9-b78e-5725aac9de1e\") "
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.924626    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-socket" pod "0016f269-b213-41a9-b78e-5725aac9de1e" (UID: "0016f269-b213-41a9-b78e-5725aac9de1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.924967    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.925123    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.925240    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m664b\" (UniqueName: \"kubernetes.io/projected/265af9b2-d3a0-465a-8200-7db6c861b0ad-kube-api-access-m664b\") pod \"maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.925413    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.925431    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:18.925969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.925446    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:18.927554 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.927147    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0016f269-b213-41a9-b78e-5725aac9de1e-istiod-ca-cert" pod "0016f269-b213-41a9-b78e-5725aac9de1e" (UID: "0016f269-b213-41a9-b78e-5725aac9de1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:18.928016 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.927694    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-data" pod "0016f269-b213-41a9-b78e-5725aac9de1e" (UID: "0016f269-b213-41a9-b78e-5725aac9de1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:18.930191 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.930143    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:18.946856 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.946802    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0016f269-b213-41a9-b78e-5725aac9de1e-istio-podinfo" pod "0016f269-b213-41a9-b78e-5725aac9de1e" (UID: "0016f269-b213-41a9-b78e-5725aac9de1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:59:18.947659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.947605    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:18.948000 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.947954    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-kube-api-access-g6lwn" pod "0016f269-b213-41a9-b78e-5725aac9de1e" (UID: "0016f269-b213-41a9-b78e-5725aac9de1e"). InnerVolumeSpecName "kube-api-access-g6lwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:18.948272 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.948232    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-istio-token" pod "0016f269-b213-41a9-b78e-5725aac9de1e" (UID: "0016f269-b213-41a9-b78e-5725aac9de1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:18.948488 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.948460    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-envoy" pod "0016f269-b213-41a9-b78e-5725aac9de1e" (UID: "0016f269-b213-41a9-b78e-5725aac9de1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:18.954022 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.952095    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m664b\" (UniqueName: \"kubernetes.io/projected/265af9b2-d3a0-465a-8200-7db6c861b0ad-kube-api-access-m664b\") pod \"maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:18.997432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.996863    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:59:18.999685 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:18.999613    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:59:19.007523 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.007493    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"]
Oct 30 23:59:19.025935 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.025900    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-podinfo\") pod \"77e0b9a5-206c-4589-a9dc-4b4f46620605\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") "
Oct 30 23:59:19.026096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.025971    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istiod-ca-cert\") pod \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") "
Oct 30 23:59:19.026096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026048    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-certs\") pod \"77e0b9a5-206c-4589-a9dc-4b4f46620605\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") "
Oct 30 23:59:19.026096 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026076    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-kube-api-access-td47r\" (UniqueName: \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-kube-api-access-td47r\") pod \"77e0b9a5-206c-4589-a9dc-4b4f46620605\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") "
Oct 30 23:59:19.026294 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026108    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-certs\") pod \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") "
Oct 30 23:59:19.026294 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026168    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-socket\") pod \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") "
Oct 30 23:59:19.026294 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026203    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-credential-socket\") pod \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") "
Oct 30 23:59:19.026294 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026236    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-envoy\") pod \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") "
Oct 30 23:59:19.026565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026298    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-podinfo\") pod \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") "
Oct 30 23:59:19.026565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026362    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-token\" (UniqueName: \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-token\") pod \"77e0b9a5-206c-4589-a9dc-4b4f46620605\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") "
Oct 30 23:59:19.026565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026489    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-credential-socket\") pod \"77e0b9a5-206c-4589-a9dc-4b4f46620605\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") "
Oct 30 23:59:19.026565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026527    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e0b9a5-206c-4589-a9dc-4b4f46620605-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e0b9a5-206c-4589-a9dc-4b4f46620605-istiod-ca-cert\") pod \"77e0b9a5-206c-4589-a9dc-4b4f46620605\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") "
Oct 30 23:59:19.026832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026578    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-token\" (UniqueName: \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-token\") pod \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") "
Oct 30 23:59:19.026832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026573    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-certs" pod "9d4c30d5-2321-45c8-8a25-ce66a86629a6" (UID: "9d4c30d5-2321-45c8-8a25-ce66a86629a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:19.026832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026633    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-socket\") pod \"77e0b9a5-206c-4589-a9dc-4b4f46620605\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") "
Oct 30 23:59:19.026832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026666    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-kube-api-access-wfpbk\" (UniqueName: \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-kube-api-access-wfpbk\") pod \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") "
Oct 30 23:59:19.026832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026740    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-envoy\") pod \"77e0b9a5-206c-4589-a9dc-4b4f46620605\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") "
Oct 30 23:59:19.026832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026775    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-data\") pod \"77e0b9a5-206c-4589-a9dc-4b4f46620605\" (UID: \"77e0b9a5-206c-4589-a9dc-4b4f46620605\") "
Oct 30 23:59:19.026832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026804    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-data\") pod \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\" (UID: \"9d4c30d5-2321-45c8-8a25-ce66a86629a6\") "
Oct 30 23:59:19.027320 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.026921    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istiod-ca-cert" pod "9d4c30d5-2321-45c8-8a25-ce66a86629a6" (UID: "9d4c30d5-2321-45c8-8a25-ce66a86629a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:19.027320 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027213    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-certs" pod "77e0b9a5-206c-4589-a9dc-4b4f46620605" (UID: "77e0b9a5-206c-4589-a9dc-4b4f46620605"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:19.027543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027485    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6lwn\" (UniqueName: \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-kube-api-access-g6lwn\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.027543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027511    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.027543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027524    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0016f269-b213-41a9-b78e-5725aac9de1e-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.027543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027538    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.027823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027553    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0016f269-b213-41a9-b78e-5725aac9de1e-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.027823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027566    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0016f269-b213-41a9-b78e-5725aac9de1e-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.027823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027580    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.027823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027594    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.027823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027607    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0016f269-b213-41a9-b78e-5725aac9de1e-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.027823 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027797    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-credential-socket" pod "77e0b9a5-206c-4589-a9dc-4b4f46620605" (UID: "77e0b9a5-206c-4589-a9dc-4b4f46620605"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:19.028146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.027973    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-socket" pod "9d4c30d5-2321-45c8-8a25-ce66a86629a6" (UID: "9d4c30d5-2321-45c8-8a25-ce66a86629a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:19.028146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.028140    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-credential-socket" pod "9d4c30d5-2321-45c8-8a25-ce66a86629a6" (UID: "9d4c30d5-2321-45c8-8a25-ce66a86629a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:19.028644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.028622    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-data" pod "9d4c30d5-2321-45c8-8a25-ce66a86629a6" (UID: "9d4c30d5-2321-45c8-8a25-ce66a86629a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:19.029161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.029135    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e0b9a5-206c-4589-a9dc-4b4f46620605-istiod-ca-cert" pod "77e0b9a5-206c-4589-a9dc-4b4f46620605" (UID: "77e0b9a5-206c-4589-a9dc-4b4f46620605"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:19.029467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.029440    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-data" pod "77e0b9a5-206c-4589-a9dc-4b4f46620605" (UID: "77e0b9a5-206c-4589-a9dc-4b4f46620605"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:19.030309 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.030283    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-socket" pod "77e0b9a5-206c-4589-a9dc-4b4f46620605" (UID: "77e0b9a5-206c-4589-a9dc-4b4f46620605"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:19.030577 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.030407    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"]
Oct 30 23:59:19.033640 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.033611    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-envoy" pod "77e0b9a5-206c-4589-a9dc-4b4f46620605" (UID: "77e0b9a5-206c-4589-a9dc-4b4f46620605"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:19.034229 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.034201    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-kube-api-access-wfpbk" pod "9d4c30d5-2321-45c8-8a25-ce66a86629a6" (UID: "9d4c30d5-2321-45c8-8a25-ce66a86629a6"). InnerVolumeSpecName "kube-api-access-wfpbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:19.034314 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.034267    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-token" pod "9d4c30d5-2321-45c8-8a25-ce66a86629a6" (UID: "9d4c30d5-2321-45c8-8a25-ce66a86629a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:19.035295 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.035276    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-token" pod "77e0b9a5-206c-4589-a9dc-4b4f46620605" (UID: "77e0b9a5-206c-4589-a9dc-4b4f46620605"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:19.037161 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.037136    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-envoy" pod "9d4c30d5-2321-45c8-8a25-ce66a86629a6" (UID: "9d4c30d5-2321-45c8-8a25-ce66a86629a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:19.037709 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.037298    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-podinfo" pod "77e0b9a5-206c-4589-a9dc-4b4f46620605" (UID: "77e0b9a5-206c-4589-a9dc-4b4f46620605"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:59:19.038969 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.038944    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-kube-api-access-td47r" pod "77e0b9a5-206c-4589-a9dc-4b4f46620605" (UID: "77e0b9a5-206c-4589-a9dc-4b4f46620605"). InnerVolumeSpecName "kube-api-access-td47r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:19.039045 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.039026    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-podinfo" pod "9d4c30d5-2321-45c8-8a25-ce66a86629a6" (UID: "9d4c30d5-2321-45c8-8a25-ce66a86629a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:59:19.047265 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.047243    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:19.128975 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.128934    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129147 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129136    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td47r\" (UniqueName: \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-kube-api-access-td47r\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129223    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129283 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129276    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129332    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129419    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129473    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129538 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129531    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129586    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e0b9a5-206c-4589-a9dc-4b4f46620605-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129646 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129640    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129702 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129694    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129759 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129751    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfpbk\" (UniqueName: \"kubernetes.io/projected/9d4c30d5-2321-45c8-8a25-ce66a86629a6-kube-api-access-wfpbk\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129811 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129804    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129858    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e0b9a5-206c-4589-a9dc-4b4f46620605-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.129917 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.129910    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d4c30d5-2321-45c8-8a25-ce66a86629a6-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:19.221790 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.221699    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b014885-3f0e-43d6-bf70-de300c1ae2db" path="/var/lib/kubelet/pods/1b014885-3f0e-43d6-bf70-de300c1ae2db/volumes"
Oct 30 23:59:19.222241 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.222222    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85fac7c9-babe-422f-96f4-d58352606a54" path="/var/lib/kubelet/pods/85fac7c9-babe-422f-96f4-d58352606a54/volumes"
Oct 30 23:59:19.222697 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.222677    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76a92f7-ee45-4e29-bac6-2d47c6557755" path="/var/lib/kubelet/pods/e76a92f7-ee45-4e29-bac6-2d47c6557755/volumes"
Oct 30 23:59:19.392627 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.392594    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"]
Oct 30 23:59:19.393152 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:59:19.393123    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265af9b2_d3a0_465a_8200_7db6c861b0ad.slice/crio-592a9ade1f76450b051bb1e47c5a2ac39c5778ae39cff969a3e20c59fde1ae7a WatchSource:0}: Error finding container 592a9ade1f76450b051bb1e47c5a2ac39c5778ae39cff969a3e20c59fde1ae7a: Status 404 returned error can't find the container with id 592a9ade1f76450b051bb1e47c5a2ac39c5778ae39cff969a3e20c59fde1ae7a
Oct 30 23:59:19.683296 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.683269    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"
Oct 30 23:59:19.683738 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.683269    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l" event={"ID":"9d4c30d5-2321-45c8-8a25-ce66a86629a6","Type":"ContainerDied","Data":"def039d0c9fd51952117b0a02cda2bc833dff145444942ac5199f442a757c0c9"}
Oct 30 23:59:19.683738 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.683463    4555 scope.go:122] "RemoveContainer" containerID="5073d300c5d7359d181fcbb6151596c94e1f7e95cebf812c49ba23068b860944"
Oct 30 23:59:19.687997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.687965    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"
Oct 30 23:59:19.687997 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.687985    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g" event={"ID":"77e0b9a5-206c-4589-a9dc-4b4f46620605","Type":"ContainerDied","Data":"51541d753b44105d7e53a7c6009b68ba026859393309f599cdd94d0c3799b59d"}
Oct 30 23:59:19.690648 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.690585    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt" event={"ID":"265af9b2-d3a0-465a-8200-7db6c861b0ad","Type":"ContainerStarted","Data":"592a9ade1f76450b051bb1e47c5a2ac39c5778ae39cff969a3e20c59fde1ae7a"}
Oct 30 23:59:19.692780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.692756    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd" event={"ID":"0016f269-b213-41a9-b78e-5725aac9de1e","Type":"ContainerDied","Data":"645c2f36677aec0d1272d0e8c0755654045032e4761b857ea8c56e0d4080933a"}
Oct 30 23:59:19.692914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.692822    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"
Oct 30 23:59:19.700011 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.699990    4555 scope.go:122] "RemoveContainer" containerID="3c135d038be4c16cba2ebf437fb8115bbc791e7d6bbe24872ad729168513ca30"
Oct 30 23:59:19.711543 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.711524    4555 scope.go:122] "RemoveContainer" containerID="e23079989e9cb653203eef06c3a11a54a11b205148409368bd4a7a8b5a74f4c2"
Oct 30 23:59:19.735777 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.735749    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"]
Oct 30 23:59:19.741971 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.741947    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-thjrd"]
Oct 30 23:59:19.769484 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.769456    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"]
Oct 30 23:59:19.777287 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.777251    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-qwv5g"]
Oct 30 23:59:19.814535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.814494    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"]
Oct 30 23:59:19.822857 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.822558    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b0299-openshift-default-5f568545c9-snb6l"]
Oct 30 23:59:19.849143 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:19.849093    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt" podStartSLOduration=1.84907753 podStartE2EDuration="1.84907753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:19.848577013 +0000 UTC m=+2418.775214594" watchObservedRunningTime="2026-10-30 23:59:19.84907753 +0000 UTC m=+2418.775715110"
Oct 30 23:59:20.536117 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:20.536081    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"]
Oct 30 23:59:20.540612 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:20.540588    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c5084-whh8r"]
Oct 30 23:59:20.709552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:20.709462    4555 generic.go:356] "Generic (PLEG): container finished" podID="265af9b2-d3a0-465a-8200-7db6c861b0ad" containerID="bb358653e89be232640b6bb291bdb6a6cef907df967f00f2d41ec5e53a4d6eb0" exitCode=0
Oct 30 23:59:20.709552 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:20.709541    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt" event={"ID":"265af9b2-d3a0-465a-8200-7db6c861b0ad","Type":"ContainerDied","Data":"bb358653e89be232640b6bb291bdb6a6cef907df967f00f2d41ec5e53a4d6eb0"}
Oct 30 23:59:21.210065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:21.210024    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" path="/var/lib/kubelet/pods/0016f269-b213-41a9-b78e-5725aac9de1e/volumes"
Oct 30 23:59:21.210604 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:21.210589    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" path="/var/lib/kubelet/pods/77e0b9a5-206c-4589-a9dc-4b4f46620605/volumes"
Oct 30 23:59:21.211028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:21.211015    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" path="/var/lib/kubelet/pods/9d4c30d5-2321-45c8-8a25-ce66a86629a6/volumes"
Oct 30 23:59:21.211842 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:21.211822    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b52db4-a62f-45ce-8630-75172184ec77" path="/var/lib/kubelet/pods/b1b52db4-a62f-45ce-8630-75172184ec77/volumes"
Oct 30 23:59:21.259099 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:21.259066    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:21.272334 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:21.272307    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:21.666358 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:21.666318    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 30 23:59:21.736511 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:21.736483    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 30 23:59:21.987251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:21.987222    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:22.024394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.015866    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265af9b2-d3a0-465a-8200-7db6c861b0ad-kube-api-access-m664b\" (UniqueName: \"kubernetes.io/projected/265af9b2-d3a0-465a-8200-7db6c861b0ad-kube-api-access-m664b\") pod \"265af9b2-d3a0-465a-8200-7db6c861b0ad\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") "
Oct 30 23:59:22.024394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.015971    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-openshift-ca\") pod \"265af9b2-d3a0-465a-8200-7db6c861b0ad\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") "
Oct 30 23:59:22.024394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.016104    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-xks-ca\") pod \"265af9b2-d3a0-465a-8200-7db6c861b0ad\" (UID: \"265af9b2-d3a0-465a-8200-7db6c861b0ad\") "
Oct 30 23:59:22.026340 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.026274    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-openshift-ca" pod "265af9b2-d3a0-465a-8200-7db6c861b0ad" (UID: "265af9b2-d3a0-465a-8200-7db6c861b0ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:22.066027 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.065950    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265af9b2-d3a0-465a-8200-7db6c861b0ad-kube-api-access-m664b" pod "265af9b2-d3a0-465a-8200-7db6c861b0ad" (UID: "265af9b2-d3a0-465a-8200-7db6c861b0ad"). InnerVolumeSpecName "kube-api-access-m664b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:22.075400 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.068889    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-xks-ca" pod "265af9b2-d3a0-465a-8200-7db6c861b0ad" (UID: "265af9b2-d3a0-465a-8200-7db6c861b0ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:22.140409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.124268    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m664b\" (UniqueName: \"kubernetes.io/projected/265af9b2-d3a0-465a-8200-7db6c861b0ad-kube-api-access-m664b\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:22.140409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.124306    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:22.140409 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.124324    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/265af9b2-d3a0-465a-8200-7db6c861b0ad-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:22.437395 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.434391    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"]
Oct 30 23:59:22.442851 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.442817    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a63288-wh9gm"]
Oct 30 23:59:22.753084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.752988    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"
Oct 30 23:59:22.756148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.755695    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt" event={"ID":"265af9b2-d3a0-465a-8200-7db6c861b0ad","Type":"ContainerDied","Data":"592a9ade1f76450b051bb1e47c5a2ac39c5778ae39cff969a3e20c59fde1ae7a"}
Oct 30 23:59:22.756148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:22.755795    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="592a9ade1f76450b051bb1e47c5a2ac39c5778ae39cff969a3e20c59fde1ae7a"
Oct 30 23:59:23.255197 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:23.246917    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d54bb7-b0bc-4fd1-be96-9082e9dea41b" path="/var/lib/kubelet/pods/29d54bb7-b0bc-4fd1-be96-9082e9dea41b/volumes"
Oct 30 23:59:24.111335 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:24.111303    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"]
Oct 30 23:59:24.125657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:24.125624    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72996a61-lz7vk"]
Oct 30 23:59:24.528200 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:24.528164    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"]
Oct 30 23:59:25.209625 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:25.209589    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee07856-194a-4473-8e18-831ed49bc93a" path="/var/lib/kubelet/pods/bee07856-194a-4473-8e18-831ed49bc93a/volumes"
Oct 30 23:59:26.039979 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.039943    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main" containerID="cri-o://d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782" gracePeriod=45
Oct 30 23:59:26.379952 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.379924    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:59:26.435575 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.435537    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9c4106-2c76-455b-9203-64bae7018a04-kube-api-access-lkvph\" (UniqueName: \"kubernetes.io/projected/5a9c4106-2c76-455b-9203-64bae7018a04-kube-api-access-lkvph\") pod \"5a9c4106-2c76-455b-9203-64bae7018a04\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") "
Oct 30 23:59:26.435748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.435700    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-home\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-home\") pod \"5a9c4106-2c76-455b-9203-64bae7018a04\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") "
Oct 30 23:59:26.435843 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.435825    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a9c4106-2c76-455b-9203-64bae7018a04-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9c4106-2c76-455b-9203-64bae7018a04-tls-certs\") pod \"5a9c4106-2c76-455b-9203-64bae7018a04\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") "
Oct 30 23:59:26.435924 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.435909    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-tmp-dir\") pod \"5a9c4106-2c76-455b-9203-64bae7018a04\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") "
Oct 30 23:59:26.436039 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.436022    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-dshm\") pod \"5a9c4106-2c76-455b-9203-64bae7018a04\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") "
Oct 30 23:59:26.436109 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.436070    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-model-cache\") pod \"5a9c4106-2c76-455b-9203-64bae7018a04\" (UID: \"5a9c4106-2c76-455b-9203-64bae7018a04\") "
Oct 30 23:59:26.436610 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.436581    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-model-cache" pod "5a9c4106-2c76-455b-9203-64bae7018a04" (UID: "5a9c4106-2c76-455b-9203-64bae7018a04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:26.437322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.437110    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-tmp-dir" pod "5a9c4106-2c76-455b-9203-64bae7018a04" (UID: "5a9c4106-2c76-455b-9203-64bae7018a04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:26.437322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.437123    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:26.437322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.437173    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-home" pod "5a9c4106-2c76-455b-9203-64bae7018a04" (UID: "5a9c4106-2c76-455b-9203-64bae7018a04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:26.438989 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.438968    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9c4106-2c76-455b-9203-64bae7018a04-kube-api-access-lkvph" pod "5a9c4106-2c76-455b-9203-64bae7018a04" (UID: "5a9c4106-2c76-455b-9203-64bae7018a04"). InnerVolumeSpecName "kube-api-access-lkvph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:26.444780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.444684    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-dshm" pod "5a9c4106-2c76-455b-9203-64bae7018a04" (UID: "5a9c4106-2c76-455b-9203-64bae7018a04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:26.444780 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.444749    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9c4106-2c76-455b-9203-64bae7018a04-tls-certs" pod "5a9c4106-2c76-455b-9203-64bae7018a04" (UID: "5a9c4106-2c76-455b-9203-64bae7018a04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:26.538858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.538814    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:26.538858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.538849    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9c4106-2c76-455b-9203-64bae7018a04-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:26.538858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.538862    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:26.539110 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.538873    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9c4106-2c76-455b-9203-64bae7018a04-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:26.539110 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.538886    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkvph\" (UniqueName: \"kubernetes.io/projected/5a9c4106-2c76-455b-9203-64bae7018a04-kube-api-access-lkvph\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:26.806218 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.806172    4555 generic.go:356] "Generic (PLEG): container finished" podID="5a9c4106-2c76-455b-9203-64bae7018a04" containerID="d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782" exitCode=0
Oct 30 23:59:26.806483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.806221    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"
Oct 30 23:59:26.806483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.806250    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" event={"ID":"5a9c4106-2c76-455b-9203-64bae7018a04","Type":"ContainerDied","Data":"d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782"}
Oct 30 23:59:26.806483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.806298    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g" event={"ID":"5a9c4106-2c76-455b-9203-64bae7018a04","Type":"ContainerDied","Data":"d0fb593bdd44adc01e98b79a131245656793e4beadccf69ebd09eb3084284004"}
Oct 30 23:59:26.806483 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.806319    4555 scope.go:122] "RemoveContainer" containerID="d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782"
Oct 30 23:59:26.818695 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.818672    4555 scope.go:122] "RemoveContainer" containerID="d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782"
Oct 30 23:59:26.819009 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:26.818990    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782\": container with ID starting with d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782 not found: ID does not exist" containerID="d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782"
Oct 30 23:59:26.819058 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.819018    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782"} err="failed to get container status \"d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782\": rpc error: code = NotFound desc = could not find container \"d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782\": container with ID starting with d5ff87c0a399149564daa8b836dd3a7fbe4a01caafd46f2ab93ee534695d0782 not found: ID does not exist"
Oct 30 23:59:26.882194 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.882156    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"]
Oct 30 23:59:26.904421 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:26.904379    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-e15304/simulator-w6-e15304-kserve-74f6db8755-68g4g"]
Oct 30 23:59:27.116066 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:27.115978    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"]
Oct 30 23:59:27.116256 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:27.116228    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy" containerID="cri-o://12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5" gracePeriod=30
Oct 30 23:59:27.209183 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:27.209148    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" path="/var/lib/kubelet/pods/5a9c4106-2c76-455b-9203-64bae7018a04/volumes"
Oct 30 23:59:27.454839 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:27.454733    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:59:28.169681 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.169633    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"]
Oct 30 23:59:28.170626 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170605    4555 cpu_manager.go:414] "containerMap: removing container" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy"
Oct 30 23:59:28.170719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170629    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy"
Oct 30 23:59:28.170719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170671    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy"
Oct 30 23:59:28.170719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170679    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy"
Oct 30 23:59:28.170719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170693    4555 cpu_manager.go:414] "containerMap: removing container" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main"
Oct 30 23:59:28.170719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170702    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main"
Oct 30 23:59:28.170719 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170714    4555 cpu_manager.go:414] "containerMap: removing container" podUID="265af9b2-d3a0-465a-8200-7db6c861b0ad" containerName="revoke-keys"
Oct 30 23:59:28.170925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170722    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265af9b2-d3a0-465a-8200-7db6c861b0ad" containerName="revoke-keys"
Oct 30 23:59:28.170925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170756    4555 cpu_manager.go:414] "containerMap: removing container" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy"
Oct 30 23:59:28.170925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170763    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy"
Oct 30 23:59:28.170925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170915    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy"
Oct 30 23:59:28.171084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170926    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9c4106-2c76-455b-9203-64bae7018a04" containerName="main"
Oct 30 23:59:28.171084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170937    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="265af9b2-d3a0-465a-8200-7db6c861b0ad" containerName="revoke-keys"
Oct 30 23:59:28.171084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170951    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d4c30d5-2321-45c8-8a25-ce66a86629a6" containerName="istio-proxy"
Oct 30 23:59:28.171084 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.170964    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e0b9a5-206c-4589-a9dc-4b4f46620605" containerName="istio-proxy"
Oct 30 23:59:28.176311 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.176286    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.190955 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.190929    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:59:28.192416 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.192390    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"]
Oct 30 23:59:28.277938 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.277894    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45zm\" (UniqueName: \"kubernetes.io/projected/a9f1254c-9f27-4f46-88f1-99650f88a59b-kube-api-access-v45zm\") pod \"maas-api-revoke-sub-d25af825-mmdfx\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.278128 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.277946    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25af825-mmdfx\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.278128 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.278067    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25af825-mmdfx\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.285471 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.285440    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:59:28.380095 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.380043    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45zm\" (UniqueName: \"kubernetes.io/projected/a9f1254c-9f27-4f46-88f1-99650f88a59b-kube-api-access-v45zm\") pod \"maas-api-revoke-sub-d25af825-mmdfx\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.380437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.380116    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25af825-mmdfx\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.380437 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.380222    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25af825-mmdfx\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.380921 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.380896    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25af825-mmdfx\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.383074 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.383047    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25af825-mmdfx\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.396270 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.396241    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45zm\" (UniqueName: \"kubernetes.io/projected/a9f1254c-9f27-4f46-88f1-99650f88a59b-kube-api-access-v45zm\") pod \"maas-api-revoke-sub-d25af825-mmdfx\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.504729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.504635    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:28.839509 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:59:28.839475    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f1254c_9f27_4f46_88f1_99650f88a59b.slice/crio-98bf59b230a9a2b7d0aa5db42e21e5c5c2d349aec0b1ddd2743b5c2b2e99b111 WatchSource:0}: Error finding container 98bf59b230a9a2b7d0aa5db42e21e5c5c2d349aec0b1ddd2743b5c2b2e99b111: Status 404 returned error can't find the container with id 98bf59b230a9a2b7d0aa5db42e21e5c5c2d349aec0b1ddd2743b5c2b2e99b111
Oct 30 23:59:28.842266 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.842238    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"]
Oct 30 23:59:28.855076 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:28.855050    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"]
Oct 30 23:59:29.529080 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:29.529038    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd" podUID="8b131cf2-9074-493b-a5c9-0ee97d47a566" containerName="payload-processing" containerID="cri-o://ad6991788b8e4184f3c0a2292586b6c53ad8c342409a8e354cc1c2cfba8839ea" gracePeriod=25
Oct 30 23:59:29.836489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:29.836383    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx" event={"ID":"a9f1254c-9f27-4f46-88f1-99650f88a59b","Type":"ContainerStarted","Data":"9ba57ef8046d95497ec2dbcc4e2c26b0b7d115507e53b939396fad9e8cb89979"}
Oct 30 23:59:29.836489 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:29.836435    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx" event={"ID":"a9f1254c-9f27-4f46-88f1-99650f88a59b","Type":"ContainerStarted","Data":"98bf59b230a9a2b7d0aa5db42e21e5c5c2d349aec0b1ddd2743b5c2b2e99b111"}
Oct 30 23:59:29.857693 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:29.857631    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx" podStartSLOduration=1.8576104230000001 podStartE2EDuration="1.857610423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:29.857504316 +0000 UTC m=+2428.784141898" watchObservedRunningTime="2026-10-30 23:59:29.857610423 +0000 UTC m=+2428.784248005"
Oct 30 23:59:30.901765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:30.899109    4555 generic.go:356] "Generic (PLEG): container finished" podID="a9f1254c-9f27-4f46-88f1-99650f88a59b" containerID="9ba57ef8046d95497ec2dbcc4e2c26b0b7d115507e53b939396fad9e8cb89979" exitCode=0
Oct 30 23:59:30.901765 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:30.899455    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx" event={"ID":"a9f1254c-9f27-4f46-88f1-99650f88a59b","Type":"ContainerDied","Data":"9ba57ef8046d95497ec2dbcc4e2c26b0b7d115507e53b939396fad9e8cb89979"}
Oct 30 23:59:31.156022 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.155980    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"]
Oct 30 23:59:31.157342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.157314    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy"
Oct 30 23:59:31.157342 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.157341    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy"
Oct 30 23:59:31.157752 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.157732    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0016f269-b213-41a9-b78e-5725aac9de1e" containerName="istio-proxy"
Oct 30 23:59:31.164162 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.164133    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.168360 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.168330    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"]
Oct 30 23:59:31.271164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.271131    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9vm\" (UniqueName: \"kubernetes.io/projected/df21cc95-0cad-4fe7-a65e-78273c166e2c-kube-api-access-cz9vm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.271413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.271175    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.271413 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.271207    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.372905 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.372868    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9vm\" (UniqueName: \"kubernetes.io/projected/df21cc95-0cad-4fe7-a65e-78273c166e2c-kube-api-access-cz9vm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.373086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.372912    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.373086 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.372938    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.373748 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.373716    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.375710 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.375688    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.387700 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.387675    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9vm\" (UniqueName: \"kubernetes.io/projected/df21cc95-0cad-4fe7-a65e-78273c166e2c-kube-api-access-cz9vm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.491597 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.491518    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:31.839040 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:59:31.839006    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf21cc95_0cad_4fe7_a65e_78273c166e2c.slice/crio-89d2e232e686fa9aa95b808dd554029e6f1c38840455c73aa94ed34865e89eaf WatchSource:0}: Error finding container 89d2e232e686fa9aa95b808dd554029e6f1c38840455c73aa94ed34865e89eaf: Status 404 returned error can't find the container with id 89d2e232e686fa9aa95b808dd554029e6f1c38840455c73aa94ed34865e89eaf
Oct 30 23:59:31.839175 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.839049    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"]
Oct 30 23:59:31.911588 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:31.911549    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9" event={"ID":"df21cc95-0cad-4fe7-a65e-78273c166e2c","Type":"ContainerStarted","Data":"89d2e232e686fa9aa95b808dd554029e6f1c38840455c73aa94ed34865e89eaf"}
Oct 30 23:59:32.090435 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.090319    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"]
Oct 30 23:59:32.099422 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.090574    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy" containerID="cri-o://aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713" gracePeriod=30
Oct 30 23:59:32.108155 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.108131    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:32.209397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.209108    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-openshift-ca\") pod \"a9f1254c-9f27-4f46-88f1-99650f88a59b\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") "
Oct 30 23:59:32.209397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.209196    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-xks-ca\") pod \"a9f1254c-9f27-4f46-88f1-99650f88a59b\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") "
Oct 30 23:59:32.209397 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.209235    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f1254c-9f27-4f46-88f1-99650f88a59b-kube-api-access-v45zm\" (UniqueName: \"kubernetes.io/projected/a9f1254c-9f27-4f46-88f1-99650f88a59b-kube-api-access-v45zm\") pod \"a9f1254c-9f27-4f46-88f1-99650f88a59b\" (UID: \"a9f1254c-9f27-4f46-88f1-99650f88a59b\") "
Oct 30 23:59:32.230353 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.230296    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-openshift-ca" pod "a9f1254c-9f27-4f46-88f1-99650f88a59b" (UID: "a9f1254c-9f27-4f46-88f1-99650f88a59b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:32.260431 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.243034    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f1254c-9f27-4f46-88f1-99650f88a59b-kube-api-access-v45zm" pod "a9f1254c-9f27-4f46-88f1-99650f88a59b" (UID: "a9f1254c-9f27-4f46-88f1-99650f88a59b"). InnerVolumeSpecName "kube-api-access-v45zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:32.277914 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.277347    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-xks-ca" pod "a9f1254c-9f27-4f46-88f1-99650f88a59b" (UID: "a9f1254c-9f27-4f46-88f1-99650f88a59b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:32.350496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.333052    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.350496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.333093    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f1254c-9f27-4f46-88f1-99650f88a59b-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.350496 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.333108    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v45zm\" (UniqueName: \"kubernetes.io/projected/a9f1254c-9f27-4f46-88f1-99650f88a59b-kube-api-access-v45zm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.536620 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.536596    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:59:32.637299 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637218    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-certs\") pod \"92725345-6154-47a7-847c-49feff85cdc7\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") "
Oct 30 23:59:32.637457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637316    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92725345-6154-47a7-847c-49feff85cdc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92725345-6154-47a7-847c-49feff85cdc7-istiod-ca-cert\") pod \"92725345-6154-47a7-847c-49feff85cdc7\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") "
Oct 30 23:59:32.637457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637349    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-data\") pod \"92725345-6154-47a7-847c-49feff85cdc7\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") "
Oct 30 23:59:32.637457 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637397    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-envoy\") pod \"92725345-6154-47a7-847c-49feff85cdc7\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") "
Oct 30 23:59:32.637595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637482    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-socket\") pod \"92725345-6154-47a7-847c-49feff85cdc7\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") "
Oct 30 23:59:32.637595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637503    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-certs" pod "92725345-6154-47a7-847c-49feff85cdc7" (UID: "92725345-6154-47a7-847c-49feff85cdc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:32.637714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637626    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-credential-socket\") pod \"92725345-6154-47a7-847c-49feff85cdc7\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") "
Oct 30 23:59:32.637714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637665    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-data" pod "92725345-6154-47a7-847c-49feff85cdc7" (UID: "92725345-6154-47a7-847c-49feff85cdc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:32.637714 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637703    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-istio-token\" (UniqueName: \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-istio-token\") pod \"92725345-6154-47a7-847c-49feff85cdc7\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") "
Oct 30 23:59:32.637865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637738    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-kube-api-access-t2wv5\" (UniqueName: \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-kube-api-access-t2wv5\") pod \"92725345-6154-47a7-847c-49feff85cdc7\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") "
Oct 30 23:59:32.637865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637769    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92725345-6154-47a7-847c-49feff85cdc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92725345-6154-47a7-847c-49feff85cdc7-istio-podinfo\") pod \"92725345-6154-47a7-847c-49feff85cdc7\" (UID: \"92725345-6154-47a7-847c-49feff85cdc7\") "
Oct 30 23:59:32.637865 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637801    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-socket" pod "92725345-6154-47a7-847c-49feff85cdc7" (UID: "92725345-6154-47a7-847c-49feff85cdc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:32.638002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637875    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-credential-socket" pod "92725345-6154-47a7-847c-49feff85cdc7" (UID: "92725345-6154-47a7-847c-49feff85cdc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:32.638002 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.637895    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92725345-6154-47a7-847c-49feff85cdc7-istiod-ca-cert" pod "92725345-6154-47a7-847c-49feff85cdc7" (UID: "92725345-6154-47a7-847c-49feff85cdc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:32.638493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.638326    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.638493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.638385    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92725345-6154-47a7-847c-49feff85cdc7-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.638493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.638399    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.638493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.638414    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.638493 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.638427    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.639860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.639834    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-envoy" pod "92725345-6154-47a7-847c-49feff85cdc7" (UID: "92725345-6154-47a7-847c-49feff85cdc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:32.640117 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.640094    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92725345-6154-47a7-847c-49feff85cdc7-istio-podinfo" pod "92725345-6154-47a7-847c-49feff85cdc7" (UID: "92725345-6154-47a7-847c-49feff85cdc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:59:32.640187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.640102    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-istio-token" pod "92725345-6154-47a7-847c-49feff85cdc7" (UID: "92725345-6154-47a7-847c-49feff85cdc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:32.640187 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.640123    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-kube-api-access-t2wv5" pod "92725345-6154-47a7-847c-49feff85cdc7" (UID: "92725345-6154-47a7-847c-49feff85cdc7"). InnerVolumeSpecName "kube-api-access-t2wv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:32.740204 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.740174    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.740380 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.740210    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2wv5\" (UniqueName: \"kubernetes.io/projected/92725345-6154-47a7-847c-49feff85cdc7-kube-api-access-t2wv5\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.740380 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.740226    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92725345-6154-47a7-847c-49feff85cdc7-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.740380 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.740236    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92725345-6154-47a7-847c-49feff85cdc7-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:32.921638 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.921534    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"
Oct 30 23:59:32.921638 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.921558    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx" event={"ID":"a9f1254c-9f27-4f46-88f1-99650f88a59b","Type":"ContainerDied","Data":"98bf59b230a9a2b7d0aa5db42e21e5c5c2d349aec0b1ddd2743b5c2b2e99b111"}
Oct 30 23:59:32.921638 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.921627    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98bf59b230a9a2b7d0aa5db42e21e5c5c2d349aec0b1ddd2743b5c2b2e99b111"
Oct 30 23:59:32.923852 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.923827    4555 generic.go:356] "Generic (PLEG): container finished" podID="df21cc95-0cad-4fe7-a65e-78273c166e2c" containerID="85357229d19b2645e5cce7408525a6714f0f135e9e364e3a9206306d1cfb464f" exitCode=0
Oct 30 23:59:32.923998 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.923940    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9" event={"ID":"df21cc95-0cad-4fe7-a65e-78273c166e2c","Type":"ContainerDied","Data":"85357229d19b2645e5cce7408525a6714f0f135e9e364e3a9206306d1cfb464f"}
Oct 30 23:59:32.925634 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.925611    4555 generic.go:356] "Generic (PLEG): container finished" podID="92725345-6154-47a7-847c-49feff85cdc7" containerID="12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5" exitCode=0
Oct 30 23:59:32.925758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.925648    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"
Oct 30 23:59:32.925758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.925708    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" event={"ID":"92725345-6154-47a7-847c-49feff85cdc7","Type":"ContainerDied","Data":"12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5"}
Oct 30 23:59:32.925758 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.925744    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f" event={"ID":"92725345-6154-47a7-847c-49feff85cdc7","Type":"ContainerDied","Data":"62cbb9167089e4bd3a3a5e54e83ee694bcaf05dc74a29f18f3a48b14279dc742"}
Oct 30 23:59:32.925860 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.925765    4555 scope.go:122] "RemoveContainer" containerID="12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5"
Oct 30 23:59:32.940423 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.940406    4555 scope.go:122] "RemoveContainer" containerID="12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5"
Oct 30 23:59:32.940702 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:32.940683    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5\": container with ID starting with 12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5 not found: ID does not exist" containerID="12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5"
Oct 30 23:59:32.940753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.940713    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5"} err="failed to get container status \"12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5\": rpc error: code = NotFound desc = could not find container \"12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5\": container with ID starting with 12cf0ed25134d9981f52ed0105a340788a1cc41387304beefeab53149ef2dab5 not found: ID does not exist"
Oct 30 23:59:32.986083 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.986055    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"]
Oct 30 23:59:32.993657 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:32.993631    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2ccd42-openshift-default-7bf78d58cf-sl47f"]
Oct 30 23:59:33.212097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:33.212006    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92725345-6154-47a7-847c-49feff85cdc7" path="/var/lib/kubelet/pods/92725345-6154-47a7-847c-49feff85cdc7/volumes"
Oct 30 23:59:33.855927 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:33.855884    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt" podUID="9ac3c3e5-7810-4bac-97c4-e57f5310e7ec" containerName="payload-pre-processing" containerID="cri-o://58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38" gracePeriod=25
Oct 30 23:59:34.035778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.035727    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main" containerID="cri-o://c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52" gracePeriod=45
Oct 30 23:59:34.065011 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.058883    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" containerID="cri-o://603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264" gracePeriod=45
Oct 30 23:59:34.093055 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.092446    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused"
Oct 30 23:59:34.204966 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.204930    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:59:34.215350 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.213678    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:34.364707 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.364669    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df21cc95-0cad-4fe7-a65e-78273c166e2c-kube-api-access-cz9vm\" (UniqueName: \"kubernetes.io/projected/df21cc95-0cad-4fe7-a65e-78273c166e2c-kube-api-access-cz9vm\") pod \"df21cc95-0cad-4fe7-a65e-78273c166e2c\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") "
Oct 30 23:59:34.365070 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.365043    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-openshift-ca\") pod \"df21cc95-0cad-4fe7-a65e-78273c166e2c\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") "
Oct 30 23:59:34.365267 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.365252    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-kube-api-access-bhwdc\" (UniqueName: \"kubernetes.io/projected/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-kube-api-access-bhwdc\") pod \"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec\" (UID: \"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec\") "
Oct 30 23:59:34.365435 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.365415    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-xks-ca\") pod \"df21cc95-0cad-4fe7-a65e-78273c166e2c\" (UID: \"df21cc95-0cad-4fe7-a65e-78273c166e2c\") "
Oct 30 23:59:34.365559 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.365544    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-plugins-config-volume\") pod \"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec\" (UID: \"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec\") "
Oct 30 23:59:34.368455 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.368397    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-openshift-ca" pod "df21cc95-0cad-4fe7-a65e-78273c166e2c" (UID: "df21cc95-0cad-4fe7-a65e-78273c166e2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:34.370146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.370123    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-plugins-config-volume" pod "9ac3c3e5-7810-4bac-97c4-e57f5310e7ec" (UID: "9ac3c3e5-7810-4bac-97c4-e57f5310e7ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:34.372118 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.372078    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df21cc95-0cad-4fe7-a65e-78273c166e2c-kube-api-access-cz9vm" pod "df21cc95-0cad-4fe7-a65e-78273c166e2c" (UID: "df21cc95-0cad-4fe7-a65e-78273c166e2c"). InnerVolumeSpecName "kube-api-access-cz9vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:34.374406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.374351    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-xks-ca" pod "df21cc95-0cad-4fe7-a65e-78273c166e2c" (UID: "df21cc95-0cad-4fe7-a65e-78273c166e2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:34.375118 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.375095    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-kube-api-access-bhwdc" pod "9ac3c3e5-7810-4bac-97c4-e57f5310e7ec" (UID: "9ac3c3e5-7810-4bac-97c4-e57f5310e7ec"). InnerVolumeSpecName "kube-api-access-bhwdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:34.441150 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.440728    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:59:34.443604 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.443578    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:59:34.468065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.467648    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz9vm\" (UniqueName: \"kubernetes.io/projected/df21cc95-0cad-4fe7-a65e-78273c166e2c-kube-api-access-cz9vm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.468065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.467681    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.468065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.467696    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhwdc\" (UniqueName: \"kubernetes.io/projected/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-kube-api-access-bhwdc\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.468065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.467711    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df21cc95-0cad-4fe7-a65e-78273c166e2c-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.468065 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.467725    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.569025 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.568992    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-dshm\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-dshm\") pod \"050aff6e-81f1-46f9-a18d-78e093d01888\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") "
Oct 30 23:59:34.569211 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569054    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-model-cache\") pod \"3b0a5c64-d800-4ce5-bfdb-743907777d34\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") "
Oct 30 23:59:34.569211 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569098    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-dshm\") pod \"3b0a5c64-d800-4ce5-bfdb-743907777d34\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") "
Oct 30 23:59:34.569211 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569125    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0a5c64-d800-4ce5-bfdb-743907777d34-kube-api-access-vslnd\" (UniqueName: \"kubernetes.io/projected/3b0a5c64-d800-4ce5-bfdb-743907777d34-kube-api-access-vslnd\") pod \"3b0a5c64-d800-4ce5-bfdb-743907777d34\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") "
Oct 30 23:59:34.569394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569259    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-tmp-dir\") pod \"3b0a5c64-d800-4ce5-bfdb-743907777d34\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") "
Oct 30 23:59:34.569394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569333    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-model-cache\") pod \"050aff6e-81f1-46f9-a18d-78e093d01888\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") "
Oct 30 23:59:34.569394 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569362    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b0a5c64-d800-4ce5-bfdb-743907777d34-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0a5c64-d800-4ce5-bfdb-743907777d34-tls-certs\") pod \"3b0a5c64-d800-4ce5-bfdb-743907777d34\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") "
Oct 30 23:59:34.569573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569428    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-model-cache" pod "3b0a5c64-d800-4ce5-bfdb-743907777d34" (UID: "3b0a5c64-d800-4ce5-bfdb-743907777d34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:34.569573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569448    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-tmp-dir\") pod \"050aff6e-81f1-46f9-a18d-78e093d01888\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") "
Oct 30 23:59:34.569573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569484    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-home\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-home\") pod \"050aff6e-81f1-46f9-a18d-78e093d01888\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") "
Oct 30 23:59:34.569573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569557    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/050aff6e-81f1-46f9-a18d-78e093d01888-tls-certs\" (UniqueName: \"kubernetes.io/secret/050aff6e-81f1-46f9-a18d-78e093d01888-tls-certs\") pod \"050aff6e-81f1-46f9-a18d-78e093d01888\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") "
Oct 30 23:59:34.569794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569597    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050aff6e-81f1-46f9-a18d-78e093d01888-kube-api-access-7bhc5\" (UniqueName: \"kubernetes.io/projected/050aff6e-81f1-46f9-a18d-78e093d01888-kube-api-access-7bhc5\") pod \"050aff6e-81f1-46f9-a18d-78e093d01888\" (UID: \"050aff6e-81f1-46f9-a18d-78e093d01888\") "
Oct 30 23:59:34.569794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569635    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-home\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-home\") pod \"3b0a5c64-d800-4ce5-bfdb-743907777d34\" (UID: \"3b0a5c64-d800-4ce5-bfdb-743907777d34\") "
Oct 30 23:59:34.569794 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569644    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-model-cache" pod "050aff6e-81f1-46f9-a18d-78e093d01888" (UID: "050aff6e-81f1-46f9-a18d-78e093d01888"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:34.569962 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569790    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-home" pod "050aff6e-81f1-46f9-a18d-78e093d01888" (UID: "050aff6e-81f1-46f9-a18d-78e093d01888"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:34.569962 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569802    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-tmp-dir" pod "050aff6e-81f1-46f9-a18d-78e093d01888" (UID: "050aff6e-81f1-46f9-a18d-78e093d01888"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:34.569962 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.569846    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-tmp-dir" pod "3b0a5c64-d800-4ce5-bfdb-743907777d34" (UID: "3b0a5c64-d800-4ce5-bfdb-743907777d34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:34.570126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.570025    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-home" pod "3b0a5c64-d800-4ce5-bfdb-743907777d34" (UID: "3b0a5c64-d800-4ce5-bfdb-743907777d34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:34.570519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.570479    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.570519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.570509    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.570519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.570524    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.570519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.570539    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.570519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.570553    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.570519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.570567    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.572688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.572659    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-dshm" pod "050aff6e-81f1-46f9-a18d-78e093d01888" (UID: "050aff6e-81f1-46f9-a18d-78e093d01888"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:34.572839 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.572774    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0a5c64-d800-4ce5-bfdb-743907777d34-kube-api-access-vslnd" pod "3b0a5c64-d800-4ce5-bfdb-743907777d34" (UID: "3b0a5c64-d800-4ce5-bfdb-743907777d34"). InnerVolumeSpecName "kube-api-access-vslnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:34.572906 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.572888    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050aff6e-81f1-46f9-a18d-78e093d01888-tls-certs" pod "050aff6e-81f1-46f9-a18d-78e093d01888" (UID: "050aff6e-81f1-46f9-a18d-78e093d01888"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:34.573101 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.573076    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b0a5c64-d800-4ce5-bfdb-743907777d34-tls-certs" pod "3b0a5c64-d800-4ce5-bfdb-743907777d34" (UID: "3b0a5c64-d800-4ce5-bfdb-743907777d34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:34.573192 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.573150    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050aff6e-81f1-46f9-a18d-78e093d01888-kube-api-access-7bhc5" pod "050aff6e-81f1-46f9-a18d-78e093d01888" (UID: "050aff6e-81f1-46f9-a18d-78e093d01888"). InnerVolumeSpecName "kube-api-access-7bhc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:34.573677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.573653    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-dshm" pod "3b0a5c64-d800-4ce5-bfdb-743907777d34" (UID: "3b0a5c64-d800-4ce5-bfdb-743907777d34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:34.672480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.672394    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0a5c64-d800-4ce5-bfdb-743907777d34-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.672480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.672424    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/050aff6e-81f1-46f9-a18d-78e093d01888-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.672480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.672436    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bhc5\" (UniqueName: \"kubernetes.io/projected/050aff6e-81f1-46f9-a18d-78e093d01888-kube-api-access-7bhc5\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.672480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.672445    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/050aff6e-81f1-46f9-a18d-78e093d01888-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.672480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.672454    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0a5c64-d800-4ce5-bfdb-743907777d34-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.672480 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.672468    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vslnd\" (UniqueName: \"kubernetes.io/projected/3b0a5c64-d800-4ce5-bfdb-743907777d34-kube-api-access-vslnd\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:34.946537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.946443    4555 generic.go:356] "Generic (PLEG): container finished" podID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerID="c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52" exitCode=0
Oct 30 23:59:34.946537 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.946492    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"
Oct 30 23:59:34.946764 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.946532    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" event={"ID":"3b0a5c64-d800-4ce5-bfdb-743907777d34","Type":"ContainerDied","Data":"c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52"}
Oct 30 23:59:34.946764 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.946582    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv" event={"ID":"3b0a5c64-d800-4ce5-bfdb-743907777d34","Type":"ContainerDied","Data":"d1c7b2212898cebbf9faf93b24d9cc06dd87b451dd14ecdbc74e766f16d9b89c"}
Oct 30 23:59:34.946764 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.946603    4555 scope.go:122] "RemoveContainer" containerID="c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52"
Oct 30 23:59:34.948988 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.948961    4555 generic.go:356] "Generic (PLEG): container finished" podID="9ac3c3e5-7810-4bac-97c4-e57f5310e7ec" containerID="58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38" exitCode=0
Oct 30 23:59:34.949146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.949017    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"
Oct 30 23:59:34.949146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.949035    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt" event={"ID":"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec","Type":"ContainerDied","Data":"58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38"}
Oct 30 23:59:34.949146 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.949117    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt" event={"ID":"9ac3c3e5-7810-4bac-97c4-e57f5310e7ec","Type":"ContainerDied","Data":"cabd5552a23cfc9ce184ad1435b107135d1392bb83291153d88859d491d9966d"}
Oct 30 23:59:34.950935 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.950916    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"
Oct 30 23:59:34.951049 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.950940    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9" event={"ID":"df21cc95-0cad-4fe7-a65e-78273c166e2c","Type":"ContainerDied","Data":"89d2e232e686fa9aa95b808dd554029e6f1c38840455c73aa94ed34865e89eaf"}
Oct 30 23:59:34.951049 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.950983    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89d2e232e686fa9aa95b808dd554029e6f1c38840455c73aa94ed34865e89eaf"
Oct 30 23:59:34.952858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.952765    4555 generic.go:356] "Generic (PLEG): container finished" podID="050aff6e-81f1-46f9-a18d-78e093d01888" containerID="603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264" exitCode=0
Oct 30 23:59:34.952858 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.952820    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"
Oct 30 23:59:34.953091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.952869    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" event={"ID":"050aff6e-81f1-46f9-a18d-78e093d01888","Type":"ContainerDied","Data":"603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264"}
Oct 30 23:59:34.953091 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.952898    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w" event={"ID":"050aff6e-81f1-46f9-a18d-78e093d01888","Type":"ContainerDied","Data":"f73551805f03af43d3c921d192e78e8bee22036f0449775161a43d4f77ae2886"}
Oct 30 23:59:34.961584 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.961492    4555 scope.go:122] "RemoveContainer" containerID="c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52"
Oct 30 23:59:34.961822 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:34.961797    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52\": container with ID starting with c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52 not found: ID does not exist" containerID="c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52"
Oct 30 23:59:34.961931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.961830    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52"} err="failed to get container status \"c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52\": rpc error: code = NotFound desc = could not find container \"c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52\": container with ID starting with c165634db7189ec5607d500e1652f9dd1da62e028b87c0363ea5230863a80c52 not found: ID does not exist"
Oct 30 23:59:34.961931 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.961855    4555 scope.go:122] "RemoveContainer" containerID="58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38"
Oct 30 23:59:34.974341 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.974325    4555 scope.go:122] "RemoveContainer" containerID="58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38"
Oct 30 23:59:34.974660 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:34.974639    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38\": container with ID starting with 58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38 not found: ID does not exist" containerID="58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38"
Oct 30 23:59:34.974746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.974668    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38"} err="failed to get container status \"58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38\": rpc error: code = NotFound desc = could not find container \"58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38\": container with ID starting with 58c563ed6054180da16633e1cbffd6687bf1c17277b39eb094f17af2bbd9ae38 not found: ID does not exist"
Oct 30 23:59:34.974746 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.974684    4555 scope.go:122] "RemoveContainer" containerID="603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264"
Oct 30 23:59:34.984339 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.984319    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"]
Oct 30 23:59:34.988994 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.988971    4555 scope.go:122] "RemoveContainer" containerID="603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264"
Oct 30 23:59:34.989302 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:34.989277    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264\": container with ID starting with 603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264 not found: ID does not exist" containerID="603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264"
Oct 30 23:59:34.989430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.989312    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264"} err="failed to get container status \"603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264\": rpc error: code = NotFound desc = could not find container \"603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264\": container with ID starting with 603ebceef7fceec97fdbfed7ab279201cc181e2fe78a8f666b122de611230264 not found: ID does not exist"
Oct 30 23:59:34.990352 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.990323    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/distinct-w1-611ef3-kserve-7bf6b7b75b-n8wdv"]
Oct 30 23:59:34.995673 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.995649    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"]
Oct 30 23:59:34.999116 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:34.999096    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f0817c-575c786967-rkgbt"]
Oct 30 23:59:35.011126 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:35.011103    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"]
Oct 30 23:59:35.016795 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:35.016773    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-611ef3/premium-w1-611ef3-kserve-5f7df9b957-hlk9w"]
Oct 30 23:59:35.209251 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:35.209168    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" path="/var/lib/kubelet/pods/050aff6e-81f1-46f9-a18d-78e093d01888/volumes"
Oct 30 23:59:35.209677 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:35.209663    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" path="/var/lib/kubelet/pods/3b0a5c64-d800-4ce5-bfdb-743907777d34/volumes"
Oct 30 23:59:35.210028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:35.210016    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac3c3e5-7810-4bac-97c4-e57f5310e7ec" path="/var/lib/kubelet/pods/9ac3c3e5-7810-4bac-97c4-e57f5310e7ec/volumes"
Oct 30 23:59:36.135796 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.135760    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd" podUID="8b131cf2-9074-493b-a5c9-0ee97d47a566" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:59:36.964872 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.964839    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"]
Oct 30 23:59:36.965763 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.965738    4555 cpu_manager.go:414] "containerMap: removing container" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main"
Oct 30 23:59:36.965925 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.965904    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main"
Oct 30 23:59:36.966050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.965930    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a9f1254c-9f27-4f46-88f1-99650f88a59b" containerName="revoke-keys"
Oct 30 23:59:36.966050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.965938    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f1254c-9f27-4f46-88f1-99650f88a59b" containerName="revoke-keys"
Oct 30 23:59:36.966050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.965960    4555 cpu_manager.go:414] "containerMap: removing container" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main"
Oct 30 23:59:36.966050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.965967    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main"
Oct 30 23:59:36.966050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966005    4555 cpu_manager.go:414] "containerMap: removing container" podUID="9ac3c3e5-7810-4bac-97c4-e57f5310e7ec" containerName="payload-pre-processing"
Oct 30 23:59:36.966050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966012    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac3c3e5-7810-4bac-97c4-e57f5310e7ec" containerName="payload-pre-processing"
Oct 30 23:59:36.966050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966026    4555 cpu_manager.go:414] "containerMap: removing container" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy"
Oct 30 23:59:36.966050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966033    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy"
Oct 30 23:59:36.966050 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966052    4555 cpu_manager.go:414] "containerMap: removing container" podUID="df21cc95-0cad-4fe7-a65e-78273c166e2c" containerName="revoke-keys"
Oct 30 23:59:36.966576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966059    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df21cc95-0cad-4fe7-a65e-78273c166e2c" containerName="revoke-keys"
Oct 30 23:59:36.966576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966253    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0a5c64-d800-4ce5-bfdb-743907777d34" containerName="main"
Oct 30 23:59:36.966576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966272    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f1254c-9f27-4f46-88f1-99650f88a59b" containerName="revoke-keys"
Oct 30 23:59:36.966576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966289    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="050aff6e-81f1-46f9-a18d-78e093d01888" containerName="main"
Oct 30 23:59:36.966576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966307    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac3c3e5-7810-4bac-97c4-e57f5310e7ec" containerName="payload-pre-processing"
Oct 30 23:59:36.966576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966318    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="92725345-6154-47a7-847c-49feff85cdc7" containerName="istio-proxy"
Oct 30 23:59:36.966576 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.966334    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="df21cc95-0cad-4fe7-a65e-78273c166e2c" containerName="revoke-keys"
Oct 30 23:59:36.971782 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:36.971748    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.005586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.005145    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfg4\" (UniqueName: \"kubernetes.io/projected/972e1861-195d-4c1b-b303-148291aa2c22-kube-api-access-7jfg4\") pod \"maas-api-revoke-sub-3b034523-4mbbl\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.005586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.005226    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b034523-4mbbl\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.005586 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.005420    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b034523-4mbbl\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.011237 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.009185    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"]
Oct 30 23:59:37.063323 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.063284    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:59:37.063503 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.063440    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 30 23:59:37.107047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.107015    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfg4\" (UniqueName: \"kubernetes.io/projected/972e1861-195d-4c1b-b303-148291aa2c22-kube-api-access-7jfg4\") pod \"maas-api-revoke-sub-3b034523-4mbbl\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.107222 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.107075    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b034523-4mbbl\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.107222 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.107189    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b034523-4mbbl\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.107982 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.107956    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b034523-4mbbl\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.110185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.110153    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b034523-4mbbl\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.124651 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.124627    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfg4\" (UniqueName: \"kubernetes.io/projected/972e1861-195d-4c1b-b303-148291aa2c22-kube-api-access-7jfg4\") pod \"maas-api-revoke-sub-3b034523-4mbbl\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.347230 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.347162    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:37.412485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.412455    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:59:37.451226 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.451104    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-credential-socket\") pod \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") "
Oct 30 23:59:37.451226 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.451212    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-envoy\") pod \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") "
Oct 30 23:59:37.451514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.451257    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-podinfo\") pod \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") "
Oct 30 23:59:37.451514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.451315    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-certs\") pod \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") "
Oct 30 23:59:37.451514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.451414    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-kube-api-access-7vd8l\" (UniqueName: \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-kube-api-access-7vd8l\") pod \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") "
Oct 30 23:59:37.451514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.451419    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-credential-socket" pod "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" (UID: "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:37.451514 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.451466    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-data\") pod \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") "
Oct 30 23:59:37.452401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.452038    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-certs" pod "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" (UID: "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:37.452401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.452049    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:37.452401 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.452245    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-data" pod "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" (UID: "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:37.456190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.455406    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-kube-api-access-7vd8l" pod "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" (UID: "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3"). InnerVolumeSpecName "kube-api-access-7vd8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:37.456190 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.456152    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-envoy" pod "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" (UID: "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:37.457696 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.456411    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-podinfo" pod "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" (UID: "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:59:37.552572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.552488    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-socket\") pod \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") "
Oct 30 23:59:37.552572 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.552544    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istiod-ca-cert\") pod \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") "
Oct 30 23:59:37.552778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.552585    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-token\" (UniqueName: \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-token\") pod \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\" (UID: \"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3\") "
Oct 30 23:59:37.552778 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.552765    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-socket" pod "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" (UID: "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:37.553003 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.552980    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istiod-ca-cert" pod "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" (UID: "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:37.553195 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.553177    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:37.553232 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.553201    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vd8l\" (UniqueName: \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-kube-api-access-7vd8l\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:37.553232 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.553212    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:37.553232 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.553221    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:37.553333 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.553231    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:37.553333 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.553245    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:37.553333 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.553259    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:37.554840 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.554820    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-token" pod "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" (UID: "9df9f953-d9bb-4fc1-b335-8548c1b4a1f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:37.654040 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.654011    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:37.659933 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.659906    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"]
Oct 30 23:59:37.660293 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:59:37.660267    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972e1861_195d_4c1b_b303_148291aa2c22.slice/crio-6bc81ebd0f1d56568e82aa51f4e483618b58c90af22346df29007a7182f287aa WatchSource:0}: Error finding container 6bc81ebd0f1d56568e82aa51f4e483618b58c90af22346df29007a7182f287aa: Status 404 returned error can't find the container with id 6bc81ebd0f1d56568e82aa51f4e483618b58c90af22346df29007a7182f287aa
Oct 30 23:59:37.770582 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:37.770545    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"]
Oct 30 23:59:38.012918 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:38.012884    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl" event={"ID":"972e1861-195d-4c1b-b303-148291aa2c22","Type":"ContainerStarted","Data":"6bc81ebd0f1d56568e82aa51f4e483618b58c90af22346df29007a7182f287aa"}
Oct 30 23:59:38.014383 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:38.014340    4555 generic.go:356] "Generic (PLEG): container finished" podID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerID="aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713" exitCode=0
Oct 30 23:59:38.014519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:38.014402    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" event={"ID":"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3","Type":"ContainerDied","Data":"aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713"}
Oct 30 23:59:38.014519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:38.014414    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"
Oct 30 23:59:38.014519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:38.014455    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj" event={"ID":"9df9f953-d9bb-4fc1-b335-8548c1b4a1f3","Type":"ContainerDied","Data":"ee1713667a0e7e2f5a069ca212073f46a0d6a62cae24d8746d3923dba6880140"}
Oct 30 23:59:38.014519 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:38.014484    4555 scope.go:122] "RemoveContainer" containerID="aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713"
Oct 30 23:59:38.055434 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:38.028438    4555 scope.go:122] "RemoveContainer" containerID="aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713"
Oct 30 23:59:38.056398 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:38.055855    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713\": container with ID starting with aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713 not found: ID does not exist" containerID="aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713"
Oct 30 23:59:38.056398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:38.055908    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713"} err="failed to get container status \"aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713\": rpc error: code = NotFound desc = could not find container \"aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713\": container with ID starting with aee1dda18de3df98f0b30811daf36ca98c4cbc86ec756ec9e39ca66a8b444713 not found: ID does not exist"
Oct 30 23:59:38.070418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:38.062628    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"]
Oct 30 23:59:38.070418 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:38.065838    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a63288-openshift-default-74bf986bf9-gxlsj"]
Oct 30 23:59:39.026905 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:39.026874    4555 generic.go:356] "Generic (PLEG): container finished" podID="972e1861-195d-4c1b-b303-148291aa2c22" containerID="dfa1c2b6c6a057e2c60426b6643732f10ea3a5134b16f3bb081c2cefa0c9ae1c" exitCode=0
Oct 30 23:59:39.027430 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:39.026972    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl" event={"ID":"972e1861-195d-4c1b-b303-148291aa2c22","Type":"ContainerDied","Data":"dfa1c2b6c6a057e2c60426b6643732f10ea3a5134b16f3bb081c2cefa0c9ae1c"}
Oct 30 23:59:39.244957 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:39.244904    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" path="/var/lib/kubelet/pods/9df9f953-d9bb-4fc1-b335-8548c1b4a1f3/volumes"
Oct 30 23:59:39.619981 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:39.619949    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"]
Oct 30 23:59:39.632239 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:39.632211    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f0817c-p6rbt"]
Oct 30 23:59:40.213935 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.213903    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:40.321966 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.321879    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-openshift-ca\") pod \"972e1861-195d-4c1b-b303-148291aa2c22\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") "
Oct 30 23:59:40.322130 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.321980    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-xks-ca\") pod \"972e1861-195d-4c1b-b303-148291aa2c22\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") "
Oct 30 23:59:40.322130 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.322042    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972e1861-195d-4c1b-b303-148291aa2c22-kube-api-access-7jfg4\" (UniqueName: \"kubernetes.io/projected/972e1861-195d-4c1b-b303-148291aa2c22-kube-api-access-7jfg4\") pod \"972e1861-195d-4c1b-b303-148291aa2c22\" (UID: \"972e1861-195d-4c1b-b303-148291aa2c22\") "
Oct 30 23:59:40.322273 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.322249    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-openshift-ca" pod "972e1861-195d-4c1b-b303-148291aa2c22" (UID: "972e1861-195d-4c1b-b303-148291aa2c22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:40.322749 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.322729    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:40.324411 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.324352    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-xks-ca" pod "972e1861-195d-4c1b-b303-148291aa2c22" (UID: "972e1861-195d-4c1b-b303-148291aa2c22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:40.324505 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.324429    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972e1861-195d-4c1b-b303-148291aa2c22-kube-api-access-7jfg4" pod "972e1861-195d-4c1b-b303-148291aa2c22" (UID: "972e1861-195d-4c1b-b303-148291aa2c22"). InnerVolumeSpecName "kube-api-access-7jfg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:40.424164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.424125    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/972e1861-195d-4c1b-b303-148291aa2c22-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:40.424164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.424158    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfg4\" (UniqueName: \"kubernetes.io/projected/972e1861-195d-4c1b-b303-148291aa2c22-kube-api-access-7jfg4\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:40.806945 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.806910    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"]
Oct 30 23:59:40.807592 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.807574    4555 cpu_manager.go:414] "containerMap: removing container" podUID="972e1861-195d-4c1b-b303-148291aa2c22" containerName="revoke-keys"
Oct 30 23:59:40.807653 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.807593    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972e1861-195d-4c1b-b303-148291aa2c22" containerName="revoke-keys"
Oct 30 23:59:40.807653 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.807608    4555 cpu_manager.go:414] "containerMap: removing container" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy"
Oct 30 23:59:40.807653 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.807613    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy"
Oct 30 23:59:40.807753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.807712    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="972e1861-195d-4c1b-b303-148291aa2c22" containerName="revoke-keys"
Oct 30 23:59:40.807753 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.807721    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df9f953-d9bb-4fc1-b335-8548c1b4a1f3" containerName="istio-proxy"
Oct 30 23:59:40.811723 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.811700    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:40.828524 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.822104    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6b38a4-openshift-default-dockercfg-rvxph\""
Oct 30 23:59:40.842420 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.835516    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"]
Oct 30 23:59:40.938347 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.938302    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-token\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:40.938556 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.938348    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istiod-ca-cert\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:40.938556 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.938455    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4h76\" (UniqueName: \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-kube-api-access-g4h76\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:40.938659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.938561    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-envoy\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:40.938659 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.938607    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-credential-socket\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:40.938760 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.938684    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-podinfo\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:40.938845 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.938825    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-certs\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:40.938949 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.938930    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-data\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:40.938990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:40.938970    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-socket\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.040510 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.040472    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-socket\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.040688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.040562    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-token\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.040688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.040587    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istiod-ca-cert\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.040688 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.040612    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4h76\" (UniqueName: \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-kube-api-access-g4h76\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.040876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.040696    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-envoy\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.040876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.040745    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-credential-socket\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.040876 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.040790    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-podinfo\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.041047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.040902    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-socket\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.041047 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.040918    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-certs\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.041156 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.041044    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-data\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.041246 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.041217    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-certs\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.041451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.041428    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-data\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.041678 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.041566    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istiod-ca-cert\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.041745 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.041687    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-credential-socket\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.043330 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.043304    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-envoy\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.043926 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.043881    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-podinfo\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.048054 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.048015    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl" event={"ID":"972e1861-195d-4c1b-b303-148291aa2c22","Type":"ContainerDied","Data":"6bc81ebd0f1d56568e82aa51f4e483618b58c90af22346df29007a7182f287aa"}
Oct 30 23:59:41.048054 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.048046    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"
Oct 30 23:59:41.048234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.048076    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc81ebd0f1d56568e82aa51f4e483618b58c90af22346df29007a7182f287aa"
Oct 30 23:59:41.055882 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.055851    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-token\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.055980 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.055948    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4h76\" (UniqueName: \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-kube-api-access-g4h76\") pod \"e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") " pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.144961 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.144921    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:41.213347 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.213312    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265af9b2-d3a0-465a-8200-7db6c861b0ad" path="/var/lib/kubelet/pods/265af9b2-d3a0-465a-8200-7db6c861b0ad/volumes"
Oct 30 23:59:41.487425 ip-10-0-134-83 kubenswrapper[4555]: W1030 23:59:41.487331    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc37fbe_1eb2_43f1_aadd_2a765ce337de.slice/crio-0eaa5635e4ca5dff66090fa7331614806fb7025c053428038100710732117c41 WatchSource:0}: Error finding container 0eaa5635e4ca5dff66090fa7331614806fb7025c053428038100710732117c41: Status 404 returned error can't find the container with id 0eaa5635e4ca5dff66090fa7331614806fb7025c053428038100710732117c41
Oct 30 23:59:41.487770 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.487704    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"]
Oct 30 23:59:41.490741 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.490702    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:59:41.490874 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.490788    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:59:41.490874 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:41.490834    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 23:59:42.061345 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:42.061311    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" event={"ID":"fdc37fbe-1eb2-43f1-aadd-2a765ce337de","Type":"ContainerStarted","Data":"04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f"}
Oct 30 23:59:42.061573 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:42.061401    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" event={"ID":"fdc37fbe-1eb2-43f1-aadd-2a765ce337de","Type":"ContainerStarted","Data":"0eaa5635e4ca5dff66090fa7331614806fb7025c053428038100710732117c41"}
Oct 30 23:59:42.145485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:42.145442    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:42.148028 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:42.147992    4555 patch_prober.go:29] interesting pod/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 23:59:42.148234 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:42.148059    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 23:59:42.805556 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:42.805523    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l" podUID="d87ec77e-7053-4a36-be37-9c7dd2fcbd8b" containerName="payload-pre-processing" containerID="cri-o://486167aefb366ea9aea5af235913543f121c2455cfec874ff4ac991dc3a964ca" gracePeriod=25
Oct 30 23:59:43.251756 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.147124    4555 patch_prober.go:29] interesting pod/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 23:59:43.251756 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.147182    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 23:59:43.339604 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.339561    4555 generic.go:356] "Generic (PLEG): container finished" podID="d87ec77e-7053-4a36-be37-9c7dd2fcbd8b" containerID="486167aefb366ea9aea5af235913543f121c2455cfec874ff4ac991dc3a964ca" exitCode=0
Oct 30 23:59:43.340182 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.339942    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l" event={"ID":"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b","Type":"ContainerDied","Data":"486167aefb366ea9aea5af235913543f121c2455cfec874ff4ac991dc3a964ca"}
Oct 30 23:59:43.383101 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.378997    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:59:43.407263 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.406527    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" podStartSLOduration=3.4065076530000002 podStartE2EDuration="3.406507653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:42.115021625 +0000 UTC m=+2441.041659207" watchObservedRunningTime="2026-10-30 23:59:43.406507653 +0000 UTC m=+2442.333145242"
Oct 30 23:59:43.524293 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.524256    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-plugins-config-volume\") pod \"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b\" (UID: \"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b\") "
Oct 30 23:59:43.524565 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.524541    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-kube-api-access-9dxpf\" (UniqueName: \"kubernetes.io/projected/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-kube-api-access-9dxpf\") pod \"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b\" (UID: \"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b\") "
Oct 30 23:59:43.524683 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.524657    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-plugins-config-volume" pod "d87ec77e-7053-4a36-be37-9c7dd2fcbd8b" (UID: "d87ec77e-7053-4a36-be37-9c7dd2fcbd8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:43.525417 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.525355    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:43.527040 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.527016    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-kube-api-access-9dxpf" pod "d87ec77e-7053-4a36-be37-9c7dd2fcbd8b" (UID: "d87ec77e-7053-4a36-be37-9c7dd2fcbd8b"). InnerVolumeSpecName "kube-api-access-9dxpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:43.627294 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.627256    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dxpf\" (UniqueName: \"kubernetes.io/projected/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b-kube-api-access-9dxpf\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:43.920995 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.906615    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"]
Oct 30 23:59:43.920995 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:43.906855    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc" podUID="73c8bb83-c123-421e-b1ab-7a9fbd44c9f9" containerName="maas-api" containerID="cri-o://ca8c5dd1ee46abf5703e74bfdce80958f3f414047e2ab1ceb29370e64f94c878" gracePeriod=30
Oct 30 23:59:44.149904 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.149855    4555 patch_prober.go:29] interesting pod/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 23:59:44.150141 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.149918    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 23:59:44.545810 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.540535    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l" event={"ID":"d87ec77e-7053-4a36-be37-9c7dd2fcbd8b","Type":"ContainerDied","Data":"7849e90fea2a266eb50f089c1e14b1b4e1e9e34c1b749e048402e7c834f313a4"}
Oct 30 23:59:44.545810 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.540607    4555 scope.go:122] "RemoveContainer" containerID="486167aefb366ea9aea5af235913543f121c2455cfec874ff4ac991dc3a964ca"
Oct 30 23:59:44.545810 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.540739    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"
Oct 30 23:59:44.561535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.557306    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"]
Oct 30 23:59:44.561535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.557383    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-611ef3-5djwc"]
Oct 30 23:59:44.707518 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.707473    4555 generic.go:356] "Generic (PLEG): container finished" podID="73c8bb83-c123-421e-b1ab-7a9fbd44c9f9" containerID="ca8c5dd1ee46abf5703e74bfdce80958f3f414047e2ab1ceb29370e64f94c878" exitCode=0
Oct 30 23:59:44.712819 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.712791    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc" event={"ID":"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9","Type":"ContainerDied","Data":"ca8c5dd1ee46abf5703e74bfdce80958f3f414047e2ab1ceb29370e64f94c878"}
Oct 30 23:59:44.760979 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.760940    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:59:44.799454 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.795826    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"]
Oct 30 23:59:44.800428 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.799997    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1bc837-6fbb775db7-5cb6l"]
Oct 30 23:59:44.908085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.906726    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-kube-api-access-r2g6q\" (UniqueName: \"kubernetes.io/projected/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-kube-api-access-r2g6q\") pod \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") "
Oct 30 23:59:44.908085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.906823    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-maas-api-tls\") pod \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") "
Oct 30 23:59:44.908085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.906858    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-openshift-service-ca\") pod \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") "
Oct 30 23:59:44.908085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.906880    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-metrics-certs\") pod \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\" (UID: \"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9\") "
Oct 30 23:59:44.908085 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.907914    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-openshift-service-ca" pod "73c8bb83-c123-421e-b1ab-7a9fbd44c9f9" (UID: "73c8bb83-c123-421e-b1ab-7a9fbd44c9f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:44.910088 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.910025    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-metrics-certs" pod "73c8bb83-c123-421e-b1ab-7a9fbd44c9f9" (UID: "73c8bb83-c123-421e-b1ab-7a9fbd44c9f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:44.910396 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.910358    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-maas-api-tls" pod "73c8bb83-c123-421e-b1ab-7a9fbd44c9f9" (UID: "73c8bb83-c123-421e-b1ab-7a9fbd44c9f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:44.910485 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:44.910423    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-kube-api-access-r2g6q" pod "73c8bb83-c123-421e-b1ab-7a9fbd44c9f9" (UID: "73c8bb83-c123-421e-b1ab-7a9fbd44c9f9"). InnerVolumeSpecName "kube-api-access-r2g6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:45.008464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.008425    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2g6q\" (UniqueName: \"kubernetes.io/projected/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-kube-api-access-r2g6q\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:45.008464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.008457    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-maas-api-tls\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:45.008464 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.008467    4555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-openshift-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:45.008959 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.008476    4555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9-metrics-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:45.146097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.146015    4555 patch_prober.go:29] interesting pod/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 23:59:45.146097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.146078    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 23:59:45.209972 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.209940    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ac5a69-e061-4847-98f5-8fae222fabb2" path="/var/lib/kubelet/pods/00ac5a69-e061-4847-98f5-8fae222fabb2/volumes"
Oct 30 23:59:45.210716 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.210688    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87ec77e-7053-4a36-be37-9c7dd2fcbd8b" path="/var/lib/kubelet/pods/d87ec77e-7053-4a36-be37-9c7dd2fcbd8b/volumes"
Oct 30 23:59:45.718322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.718280    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc" event={"ID":"73c8bb83-c123-421e-b1ab-7a9fbd44c9f9","Type":"ContainerDied","Data":"966455bd055025c9be200b769a1c95965efb5e4cffeb16dcd84dc2e692956703"}
Oct 30 23:59:45.718531 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.718361    4555 scope.go:122] "RemoveContainer" containerID="ca8c5dd1ee46abf5703e74bfdce80958f3f414047e2ab1ceb29370e64f94c878"
Oct 30 23:59:45.718531 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.718475    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"
Oct 30 23:59:45.752279 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.752250    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"]
Oct 30 23:59:45.781404 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:45.780117    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-611ef3-7599bbdcd8-t9vwc"]
Oct 30 23:59:46.136064 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:46.136032    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd" podUID="8b131cf2-9074-493b-a5c9-0ee97d47a566" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:59:46.145701 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:46.145672    4555 patch_prober.go:29] interesting pod/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 23:59:46.145831 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:46.145716    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 23:59:47.146832 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:47.146799    4555 patch_prober.go:29] interesting pod/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 23:59:47.147500 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:47.146870    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 23:59:47.210467 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:47.210439    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c8bb83-c123-421e-b1ab-7a9fbd44c9f9" path="/var/lib/kubelet/pods/73c8bb83-c123-421e-b1ab-7a9fbd44c9f9/volumes"
Oct 30 23:59:48.146153 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:48.146120    4555 patch_prober.go:29] interesting pod/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 23:59:48.146432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:48.146407    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 23:59:49.150535 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.150509    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:49.151140 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.151117    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:49.151824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.151806    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 30 23:59:49.778628 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.778591    4555 generic.go:356] "Generic (PLEG): container finished" podID="8b131cf2-9074-493b-a5c9-0ee97d47a566" containerID="ad6991788b8e4184f3c0a2292586b6c53ad8c342409a8e354cc1c2cfba8839ea" exitCode=0
Oct 30 23:59:49.778827 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.778664    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd" event={"ID":"8b131cf2-9074-493b-a5c9-0ee97d47a566","Type":"ContainerDied","Data":"ad6991788b8e4184f3c0a2292586b6c53ad8c342409a8e354cc1c2cfba8839ea"}
Oct 30 23:59:49.818432 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.818407    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:59:49.843495 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.843467    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b131cf2-9074-493b-a5c9-0ee97d47a566-kube-api-access-zm4zm\" (UniqueName: \"kubernetes.io/projected/8b131cf2-9074-493b-a5c9-0ee97d47a566-kube-api-access-zm4zm\") pod \"8b131cf2-9074-493b-a5c9-0ee97d47a566\" (UID: \"8b131cf2-9074-493b-a5c9-0ee97d47a566\") "
Oct 30 23:59:49.843732 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.843540    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b131cf2-9074-493b-a5c9-0ee97d47a566-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b131cf2-9074-493b-a5c9-0ee97d47a566-plugins-config-volume\") pod \"8b131cf2-9074-493b-a5c9-0ee97d47a566\" (UID: \"8b131cf2-9074-493b-a5c9-0ee97d47a566\") "
Oct 30 23:59:49.845786 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.845721    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b131cf2-9074-493b-a5c9-0ee97d47a566-plugins-config-volume" pod "8b131cf2-9074-493b-a5c9-0ee97d47a566" (UID: "8b131cf2-9074-493b-a5c9-0ee97d47a566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:49.863970 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.848727    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b131cf2-9074-493b-a5c9-0ee97d47a566-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:49.863970 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.860128    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b131cf2-9074-493b-a5c9-0ee97d47a566-kube-api-access-zm4zm" pod "8b131cf2-9074-493b-a5c9-0ee97d47a566" (UID: "8b131cf2-9074-493b-a5c9-0ee97d47a566"). InnerVolumeSpecName "kube-api-access-zm4zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:49.950164 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:49.950128    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm4zm\" (UniqueName: \"kubernetes.io/projected/8b131cf2-9074-493b-a5c9-0ee97d47a566-kube-api-access-zm4zm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:50.790185 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:50.790152    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"
Oct 30 23:59:50.790729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:50.790153    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd" event={"ID":"8b131cf2-9074-493b-a5c9-0ee97d47a566","Type":"ContainerDied","Data":"c24d1db942803c77a8c4cea953c14dcdddd83d3b82b806f12b45e7fe5e9721a3"}
Oct 30 23:59:50.790729 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:50.790321    4555 scope.go:122] "RemoveContainer" containerID="ad6991788b8e4184f3c0a2292586b6c53ad8c342409a8e354cc1c2cfba8839ea"
Oct 30 23:59:50.822968 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:50.822937    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"]
Oct 30 23:59:50.826465 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:50.826438    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-611ef3-ddd5f7f85-479sd"]
Oct 30 23:59:51.209214 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:51.209128    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b131cf2-9074-493b-a5c9-0ee97d47a566" path="/var/lib/kubelet/pods/8b131cf2-9074-493b-a5c9-0ee97d47a566/volumes"
Oct 30 23:59:51.848556 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:51.848505    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"]
Oct 30 23:59:51.853148 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:51.853124    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e15304-xlws4"]
Oct 30 23:59:52.401269 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:52.401235    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"]
Oct 30 23:59:52.405090 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:52.405063    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a3e876-qxjs9"]
Oct 30 23:59:53.207595 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:53.207557    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e88751-3dea-4c24-9bd0-ff016933c46f" path="/var/lib/kubelet/pods/63e88751-3dea-4c24-9bd0-ff016933c46f/volumes"
Oct 30 23:59:53.208103 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:53.208045    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df21cc95-0cad-4fe7-a65e-78273c166e2c" path="/var/lib/kubelet/pods/df21cc95-0cad-4fe7-a65e-78273c166e2c/volumes"
Oct 30 23:59:53.414475 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:53.414283    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"]
Oct 30 23:59:53.414757 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:53.414730    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerName="istio-proxy" containerID="cri-o://7c168374d46d34653ef98ca3fd60fc14dfe3eb2207e8d2b6c6aaafee39407e17" gracePeriod=30
Oct 30 23:59:54.397638 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:54.397597    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"]
Oct 30 23:59:54.398163 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:54.397880    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy" containerID="cri-o://49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5" gracePeriod=30
Oct 30 23:59:57.446614 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:57.446574    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 30 23:59:58.876431 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:58.876363    4555 generic.go:356] "Generic (PLEG): container finished" podID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerID="7c168374d46d34653ef98ca3fd60fc14dfe3eb2207e8d2b6c6aaafee39407e17" exitCode=0
Oct 30 23:59:58.876824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:58.876445    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" event={"ID":"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531","Type":"ContainerDied","Data":"7c168374d46d34653ef98ca3fd60fc14dfe3eb2207e8d2b6c6aaafee39407e17"}
Oct 30 23:59:58.876824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:58.876546    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29" event={"ID":"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531","Type":"ContainerDied","Data":"4ec7df2a77f5021c510f156e15df99311a88c3688007e1eb6a610c928ee34dc7"}
Oct 30 23:59:58.876824 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:58.876570    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ec7df2a77f5021c510f156e15df99311a88c3688007e1eb6a610c928ee34dc7"
Oct 30 23:59:58.877332 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:58.877053    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:59:59.037419 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.037385    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-data\") pod \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") "
Oct 30 23:59:59.038097 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.038074    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-envoy\") pod \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") "
Oct 30 23:59:59.038710 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.038689    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-credential-socket\") pod \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") "
Oct 30 23:59:59.039451 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.039433    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-token\" (UniqueName: \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-token\") pod \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") "
Oct 30 23:59:59.039898 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.039882    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-certs\") pod \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") "
Oct 30 23:59:59.040325 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.040309    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-socket\") pod \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") "
Oct 30 23:59:59.040776 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.040758    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-podinfo\") pod \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") "
Oct 30 23:59:59.041266 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.041248    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istiod-ca-cert\") pod \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") "
Oct 30 23:59:59.041900 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.041878    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-kube-api-access-75f9k\" (UniqueName: \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-kube-api-access-75f9k\") pod \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\" (UID: \"b6e0aef4-ac27-427f-85ac-f8c4fe2ba531\") "
Oct 30 23:59:59.042990 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.037979    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-data" pod "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" (UID: "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:59.043124 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.039175    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-credential-socket" pod "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" (UID: "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:59.043213 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.040239    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-certs" pod "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" (UID: "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:59.043328 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.040631    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-socket" pod "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" (UID: "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:59.043571 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.041588    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-envoy" pod "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" (UID: "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:59.043672 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.041793    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istiod-ca-cert" pod "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" (UID: "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:59.051119 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.051064    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-podinfo" pod "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" (UID: "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:59:59.052188 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.052144    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-kube-api-access-75f9k" pod "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" (UID: "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531"). InnerVolumeSpecName "kube-api-access-75f9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:59.052570 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.052535    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-token" pod "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" (UID: "b6e0aef4-ac27-427f-85ac-f8c4fe2ba531"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:59.057644 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.057623    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"]
Oct 30 23:59:59.144440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.144405    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:59.144440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.144435    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:59.144440 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.144447    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75f9k\" (UniqueName: \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-kube-api-access-75f9k\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:59.144670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.144458    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:59.144670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.144468    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:59.144670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.144477    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:59.144670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.144485    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:59.144670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.144494    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:59.144670 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.144503    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:59:59.806398 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.800180    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:59:59.929348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.927578    4555 generic.go:356] "Generic (PLEG): container finished" podID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerID="49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5" exitCode=0
Oct 30 23:59:59.929348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.927701    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"
Oct 30 23:59:59.929348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.928515    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"
Oct 30 23:59:59.929348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.928660    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" event={"ID":"e65b666d-2a91-451a-9da2-3cdf0a99be7a","Type":"ContainerDied","Data":"49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5"}
Oct 30 23:59:59.929348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.928704    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s" event={"ID":"e65b666d-2a91-451a-9da2-3cdf0a99be7a","Type":"ContainerDied","Data":"540cb91af2cfff53e63b6a136df574acb45f556575dfe3b2348072da091edc56"}
Oct 30 23:59:59.929348 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.928732    4555 scope.go:122] "RemoveContainer" containerID="49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5"
Oct 30 23:59:59.966996 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.964710    4555 scope.go:122] "RemoveContainer" containerID="49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5"
Oct 30 23:59:59.966996 ip-10-0-134-83 kubenswrapper[4555]: E1030 23:59:59.965940    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5\": container with ID starting with 49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5 not found: ID does not exist" containerID="49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5"
Oct 30 23:59:59.966996 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.965975    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5"} err="failed to get container status \"49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5\": rpc error: code = NotFound desc = could not find container \"49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5\": container with ID starting with 49e49bae0e2bc3586929d8f7d49a913156baf841699f9291e48872b5b8160dd5 not found: ID does not exist"
Oct 30 23:59:59.990406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.989147    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-socket\") pod \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") "
Oct 30 23:59:59.990406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.989304    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-credential-socket\") pod \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") "
Oct 30 23:59:59.990406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.989415    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-envoy\") pod \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") "
Oct 30 23:59:59.990406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.989448    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istiod-ca-cert\") pod \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") "
Oct 30 23:59:59.990406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.989487    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-token\" (UniqueName: \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-token\") pod \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") "
Oct 30 23:59:59.990406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.989556    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-certs\") pod \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") "
Oct 30 23:59:59.990406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.989602    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-kube-api-access-496df\" (UniqueName: \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-kube-api-access-496df\") pod \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") "
Oct 30 23:59:59.990406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.989661    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-podinfo\") pod \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") "
Oct 30 23:59:59.990406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.989691    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-data\") pod \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\" (UID: \"e65b666d-2a91-451a-9da2-3cdf0a99be7a\") "
Oct 30 23:59:59.990406 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.989969    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"]
Oct 30 23:59:59.991056 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.990784    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istiod-ca-cert" pod "e65b666d-2a91-451a-9da2-3cdf0a99be7a" (UID: "e65b666d-2a91-451a-9da2-3cdf0a99be7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:59:59.991056 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.991027    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-socket" pod "e65b666d-2a91-451a-9da2-3cdf0a99be7a" (UID: "e65b666d-2a91-451a-9da2-3cdf0a99be7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:59.991611 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.991537    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-data" pod "e65b666d-2a91-451a-9da2-3cdf0a99be7a" (UID: "e65b666d-2a91-451a-9da2-3cdf0a99be7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:59.992322 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.992298    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-certs" pod "e65b666d-2a91-451a-9da2-3cdf0a99be7a" (UID: "e65b666d-2a91-451a-9da2-3cdf0a99be7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:59:59.992562 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.992542    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-credential-socket" pod "e65b666d-2a91-451a-9da2-3cdf0a99be7a" (UID: "e65b666d-2a91-451a-9da2-3cdf0a99be7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:00.000015 ip-10-0-134-83 kubenswrapper[4555]: I1030 23:59:59.999990    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1bc837-openshift-default-7f796c9dd5-65d29"]
Oct 31 00:00:00.007773 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.007715    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-podinfo" pod "e65b666d-2a91-451a-9da2-3cdf0a99be7a" (UID: "e65b666d-2a91-451a-9da2-3cdf0a99be7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:00:00.007929 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.007776    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-envoy" pod "e65b666d-2a91-451a-9da2-3cdf0a99be7a" (UID: "e65b666d-2a91-451a-9da2-3cdf0a99be7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:00.012982 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.012900    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-kube-api-access-496df" pod "e65b666d-2a91-451a-9da2-3cdf0a99be7a" (UID: "e65b666d-2a91-451a-9da2-3cdf0a99be7a"). InnerVolumeSpecName "kube-api-access-496df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:00.012982 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.012958    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-token" pod "e65b666d-2a91-451a-9da2-3cdf0a99be7a" (UID: "e65b666d-2a91-451a-9da2-3cdf0a99be7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:00.091698 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.091656    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:00.091698 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.091693    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:00.091945 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.091708    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:00.091945 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.091722    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:00.091945 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.091740    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:00.091945 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.091753    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:00.091945 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.091767    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:00.091945 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.091779    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65b666d-2a91-451a-9da2-3cdf0a99be7a-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:00.091945 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.091794    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-496df\" (UniqueName: \"kubernetes.io/projected/e65b666d-2a91-451a-9da2-3cdf0a99be7a-kube-api-access-496df\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:00.215288 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215193    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"]
Oct 31 00:00:00.215781 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215763    4555 cpu_manager.go:414] "containerMap: removing container" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerName="istio-proxy"
Oct 31 00:00:00.215781 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215779    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerName="istio-proxy"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215796    4555 cpu_manager.go:414] "containerMap: removing container" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215802    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215817    4555 cpu_manager.go:414] "containerMap: removing container" podUID="8b131cf2-9074-493b-a5c9-0ee97d47a566" containerName="payload-processing"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215822    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b131cf2-9074-493b-a5c9-0ee97d47a566" containerName="payload-processing"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215844    4555 cpu_manager.go:414] "containerMap: removing container" podUID="d87ec77e-7053-4a36-be37-9c7dd2fcbd8b" containerName="payload-pre-processing"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215851    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87ec77e-7053-4a36-be37-9c7dd2fcbd8b" containerName="payload-pre-processing"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215863    4555 cpu_manager.go:414] "containerMap: removing container" podUID="73c8bb83-c123-421e-b1ab-7a9fbd44c9f9" containerName="maas-api"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215868    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c8bb83-c123-421e-b1ab-7a9fbd44c9f9" containerName="maas-api"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215946    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b131cf2-9074-493b-a5c9-0ee97d47a566" containerName="payload-processing"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215955    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87ec77e-7053-4a36-be37-9c7dd2fcbd8b" containerName="payload-pre-processing"
Oct 31 00:00:00.215962 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215962    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c8bb83-c123-421e-b1ab-7a9fbd44c9f9" containerName="maas-api"
Oct 31 00:00:00.216577 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215973    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" containerName="istio-proxy"
Oct 31 00:00:00.216577 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.215980    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" containerName="istio-proxy"
Oct 31 00:00:00.219327 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.219305    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"
Oct 31 00:00:00.237081 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.237038    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 31 00:00:00.309395 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.307532    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxwh\" (UniqueName: \"kubernetes.io/projected/0732ac8c-6272-46d5-950b-c5d92a2216cc-kube-api-access-qjxwh\") pod \"maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55\" (UID: \"0732ac8c-6272-46d5-950b-c5d92a2216cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"
Oct 31 00:00:00.309395 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.307701    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0732ac8c-6272-46d5-950b-c5d92a2216cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55\" (UID: \"0732ac8c-6272-46d5-950b-c5d92a2216cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"
Oct 31 00:00:00.357431 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.351432    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"]
Oct 31 00:00:00.382396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.381655    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"]
Oct 31 00:00:00.462611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.390650    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"
Oct 31 00:00:00.462611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.414001    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxwh\" (UniqueName: \"kubernetes.io/projected/0732ac8c-6272-46d5-950b-c5d92a2216cc-kube-api-access-qjxwh\") pod \"maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55\" (UID: \"0732ac8c-6272-46d5-950b-c5d92a2216cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"
Oct 31 00:00:00.462611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.414137    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0732ac8c-6272-46d5-950b-c5d92a2216cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55\" (UID: \"0732ac8c-6272-46d5-950b-c5d92a2216cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"
Oct 31 00:00:00.462611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.414181    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0618ce43-2b19-49bb-9dd5-6813e8755b84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc\" (UID: \"0618ce43-2b19-49bb-9dd5-6813e8755b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"
Oct 31 00:00:00.462611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.414227    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkx9q\" (UniqueName: \"kubernetes.io/projected/0618ce43-2b19-49bb-9dd5-6813e8755b84-kube-api-access-lkx9q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc\" (UID: \"0618ce43-2b19-49bb-9dd5-6813e8755b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"
Oct 31 00:00:00.462611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.415420    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0732ac8c-6272-46d5-950b-c5d92a2216cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55\" (UID: \"0732ac8c-6272-46d5-950b-c5d92a2216cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"
Oct 31 00:00:00.462611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.451562    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"]
Oct 31 00:00:00.489461 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.468897    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"]
Oct 31 00:00:00.489461 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.487619    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"
Oct 31 00:00:00.496396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.489902    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"
Oct 31 00:00:00.517394 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.515864    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0618ce43-2b19-49bb-9dd5-6813e8755b84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc\" (UID: \"0618ce43-2b19-49bb-9dd5-6813e8755b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"
Oct 31 00:00:00.517394 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.515933    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkx9q\" (UniqueName: \"kubernetes.io/projected/0618ce43-2b19-49bb-9dd5-6813e8755b84-kube-api-access-lkx9q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc\" (UID: \"0618ce43-2b19-49bb-9dd5-6813e8755b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"
Oct 31 00:00:00.519989 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.518449    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0618ce43-2b19-49bb-9dd5-6813e8755b84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc\" (UID: \"0618ce43-2b19-49bb-9dd5-6813e8755b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"
Oct 31 00:00:00.520947 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.520915    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"]
Oct 31 00:00:00.527489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.527459    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxwh\" (UniqueName: \"kubernetes.io/projected/0732ac8c-6272-46d5-950b-c5d92a2216cc-kube-api-access-qjxwh\") pod \"maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55\" (UID: \"0732ac8c-6272-46d5-950b-c5d92a2216cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"
Oct 31 00:00:00.529227 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.529199    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-611ef3-openshift-default-68b9548c67-klw8s"]
Oct 31 00:00:00.538502 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.538017    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"
Oct 31 00:00:00.540099 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.540066    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkx9q\" (UniqueName: \"kubernetes.io/projected/0618ce43-2b19-49bb-9dd5-6813e8755b84-kube-api-access-lkx9q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc\" (UID: \"0618ce43-2b19-49bb-9dd5-6813e8755b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"
Oct 31 00:00:00.540283 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.540165    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"]
Oct 31 00:00:00.544594 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.544560    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"]
Oct 31 00:00:00.555249 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.555210    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"]
Oct 31 00:00:00.617160 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.617127    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j489b\" (UniqueName: \"kubernetes.io/projected/727002c6-942f-4ea0-b31e-74acc34261d6-kube-api-access-j489b\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q\" (UID: \"727002c6-942f-4ea0-b31e-74acc34261d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"
Oct 31 00:00:00.617355 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.617177    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79fhk\" (UniqueName: \"kubernetes.io/projected/d3cac855-9e22-4621-a04c-884e002ead44-kube-api-access-79fhk\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j\" (UID: \"d3cac855-9e22-4621-a04c-884e002ead44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"
Oct 31 00:00:00.617355 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.617302    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727002c6-942f-4ea0-b31e-74acc34261d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q\" (UID: \"727002c6-942f-4ea0-b31e-74acc34261d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"
Oct 31 00:00:00.617505 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.617414    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3cac855-9e22-4621-a04c-884e002ead44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j\" (UID: \"d3cac855-9e22-4621-a04c-884e002ead44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"
Oct 31 00:00:00.634707 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.634679    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"
Oct 31 00:00:00.720735 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.720148    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j489b\" (UniqueName: \"kubernetes.io/projected/727002c6-942f-4ea0-b31e-74acc34261d6-kube-api-access-j489b\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q\" (UID: \"727002c6-942f-4ea0-b31e-74acc34261d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"
Oct 31 00:00:00.720735 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.720232    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79fhk\" (UniqueName: \"kubernetes.io/projected/d3cac855-9e22-4621-a04c-884e002ead44-kube-api-access-79fhk\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j\" (UID: \"d3cac855-9e22-4621-a04c-884e002ead44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"
Oct 31 00:00:00.720735 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.720416    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727002c6-942f-4ea0-b31e-74acc34261d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q\" (UID: \"727002c6-942f-4ea0-b31e-74acc34261d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"
Oct 31 00:00:00.720735 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.720485    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3cac855-9e22-4621-a04c-884e002ead44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j\" (UID: \"d3cac855-9e22-4621-a04c-884e002ead44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"
Oct 31 00:00:00.721967 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.721514    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3cac855-9e22-4621-a04c-884e002ead44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j\" (UID: \"d3cac855-9e22-4621-a04c-884e002ead44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"
Oct 31 00:00:00.726203 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.722551    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727002c6-942f-4ea0-b31e-74acc34261d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q\" (UID: \"727002c6-942f-4ea0-b31e-74acc34261d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"
Oct 31 00:00:00.747090 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.745192    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79fhk\" (UniqueName: \"kubernetes.io/projected/d3cac855-9e22-4621-a04c-884e002ead44-kube-api-access-79fhk\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j\" (UID: \"d3cac855-9e22-4621-a04c-884e002ead44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"
Oct 31 00:00:00.749851 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.748307    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j489b\" (UniqueName: \"kubernetes.io/projected/727002c6-942f-4ea0-b31e-74acc34261d6-kube-api-access-j489b\") pod \"maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q\" (UID: \"727002c6-942f-4ea0-b31e-74acc34261d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"
Oct 31 00:00:00.925465 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.925429    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"]
Oct 31 00:00:00.929238 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:00:00.929204    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0732ac8c_6272_46d5_950b_c5d92a2216cc.slice/crio-b407ade22688d2047858ae0202ed5641f96d2efeaecfcb2f77e3b43f7af366db WatchSource:0}: Error finding container b407ade22688d2047858ae0202ed5641f96d2efeaecfcb2f77e3b43f7af366db: Status 404 returned error can't find the container with id b407ade22688d2047858ae0202ed5641f96d2efeaecfcb2f77e3b43f7af366db
Oct 31 00:00:00.941571 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.941535    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"
Oct 31 00:00:00.950424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.950319    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"
Oct 31 00:00:00.978528 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:00.977670    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"]
Oct 31 00:00:00.984641 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:00:00.984605    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0618ce43_2b19_49bb_9dd5_6813e8755b84.slice/crio-b6ca65b6e6928b36327d7f9d4045b4438f45bbd78d7ccd9720b559bf3a2dbcc5 WatchSource:0}: Error finding container b6ca65b6e6928b36327d7f9d4045b4438f45bbd78d7ccd9720b559bf3a2dbcc5: Status 404 returned error can't find the container with id b6ca65b6e6928b36327d7f9d4045b4438f45bbd78d7ccd9720b559bf3a2dbcc5
Oct 31 00:00:01.210950 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.210912    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e0aef4-ac27-427f-85ac-f8c4fe2ba531" path="/var/lib/kubelet/pods/b6e0aef4-ac27-427f-85ac-f8c4fe2ba531/volumes"
Oct 31 00:00:01.211594 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.211577    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65b666d-2a91-451a-9da2-3cdf0a99be7a" path="/var/lib/kubelet/pods/e65b666d-2a91-451a-9da2-3cdf0a99be7a/volumes"
Oct 31 00:00:01.299623 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:00:01.299593    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cac855_9e22_4621_a04c_884e002ead44.slice/crio-c1edad4a4051f4c3c3af72e8f354cc8800dc0deb35da471dfd4b086378c28271 WatchSource:0}: Error finding container c1edad4a4051f4c3c3af72e8f354cc8800dc0deb35da471dfd4b086378c28271: Status 404 returned error can't find the container with id c1edad4a4051f4c3c3af72e8f354cc8800dc0deb35da471dfd4b086378c28271
Oct 31 00:00:01.302224 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:00:01.302199    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727002c6_942f_4ea0_b31e_74acc34261d6.slice/crio-4c46fe9150540a23acdc6b57ac92bdfd7c0b50d6951d92b763de5b0b24e581ee WatchSource:0}: Error finding container 4c46fe9150540a23acdc6b57ac92bdfd7c0b50d6951d92b763de5b0b24e581ee: Status 404 returned error can't find the container with id 4c46fe9150540a23acdc6b57ac92bdfd7c0b50d6951d92b763de5b0b24e581ee
Oct 31 00:00:01.305019 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.304829    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"]
Oct 31 00:00:01.310105 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.310081    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"]
Oct 31 00:00:01.946850 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.946714    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" event={"ID":"d3cac855-9e22-4621-a04c-884e002ead44","Type":"ContainerStarted","Data":"094796671b2bd701aba87b203e4f5e06f5ba6d3dbf91f85e5af04f3717c6853b"}
Oct 31 00:00:01.946850 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.946781    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" event={"ID":"d3cac855-9e22-4621-a04c-884e002ead44","Type":"ContainerStarted","Data":"c1edad4a4051f4c3c3af72e8f354cc8800dc0deb35da471dfd4b086378c28271"}
Oct 31 00:00:01.948774 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.948741    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" event={"ID":"0618ce43-2b19-49bb-9dd5-6813e8755b84","Type":"ContainerStarted","Data":"9545d42611e3d4c12689b8d357fbb14ed2387015f2ac1099adf0964c288f385e"}
Oct 31 00:00:01.948901 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.948796    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" event={"ID":"0618ce43-2b19-49bb-9dd5-6813e8755b84","Type":"ContainerStarted","Data":"b6ca65b6e6928b36327d7f9d4045b4438f45bbd78d7ccd9720b559bf3a2dbcc5"}
Oct 31 00:00:01.950407 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.950364    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q" event={"ID":"727002c6-942f-4ea0-b31e-74acc34261d6","Type":"ContainerStarted","Data":"4aae15135ae57fe6da01e0747d2328551a4d1c64debbf75c95108656306e51e8"}
Oct 31 00:00:01.950512 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.950430    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q" event={"ID":"727002c6-942f-4ea0-b31e-74acc34261d6","Type":"ContainerStarted","Data":"4c46fe9150540a23acdc6b57ac92bdfd7c0b50d6951d92b763de5b0b24e581ee"}
Oct 31 00:00:01.952103 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.952077    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" event={"ID":"0732ac8c-6272-46d5-950b-c5d92a2216cc","Type":"ContainerStarted","Data":"6fd44e83660ce3b1e6a0dc1570672098f3c22f61c59d83e40dbf7b59b8a11364"}
Oct 31 00:00:01.952217 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.952120    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" event={"ID":"0732ac8c-6272-46d5-950b-c5d92a2216cc","Type":"ContainerStarted","Data":"b407ade22688d2047858ae0202ed5641f96d2efeaecfcb2f77e3b43f7af366db"}
Oct 31 00:00:01.979935 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:01.979887    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" podStartSLOduration=1.9798716330000001 podStartE2EDuration="1.979871633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.973129293 +0000 UTC m=+2460.899766875" watchObservedRunningTime="2026-10-31 00:00:01.979871633 +0000 UTC m=+2460.906509244"
Oct 31 00:00:02.007121 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.007069    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" podStartSLOduration=2.007055088 podStartE2EDuration="2.007055088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:02.003861101 +0000 UTC m=+2460.930498681" watchObservedRunningTime="2026-10-31 00:00:02.007055088 +0000 UTC m=+2460.933692669"
Oct 31 00:00:02.032907 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.032850    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" podStartSLOduration=2.03283246 podStartE2EDuration="2.03283246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:02.031922559 +0000 UTC m=+2460.958560141" watchObservedRunningTime="2026-10-31 00:00:02.03283246 +0000 UTC m=+2460.959470041"
Oct 31 00:00:02.068886 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.068830    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q" podStartSLOduration=2.068814662 podStartE2EDuration="2.068814662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:02.064511981 +0000 UTC m=+2460.991149569" watchObservedRunningTime="2026-10-31 00:00:02.068814662 +0000 UTC m=+2460.995452243"
Oct 31 00:00:02.963029 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.962995    4555 generic.go:356] "Generic (PLEG): container finished" podID="727002c6-942f-4ea0-b31e-74acc34261d6" containerID="4aae15135ae57fe6da01e0747d2328551a4d1c64debbf75c95108656306e51e8" exitCode=60
Oct 31 00:00:02.963563 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.963080    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q" event={"ID":"727002c6-942f-4ea0-b31e-74acc34261d6","Type":"ContainerDied","Data":"4aae15135ae57fe6da01e0747d2328551a4d1c64debbf75c95108656306e51e8"}
Oct 31 00:00:02.963563 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.963421    4555 scope.go:122] "RemoveContainer" containerID="4aae15135ae57fe6da01e0747d2328551a4d1c64debbf75c95108656306e51e8"
Oct 31 00:00:02.967455 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.965778    4555 generic.go:356] "Generic (PLEG): container finished" podID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerID="6fd44e83660ce3b1e6a0dc1570672098f3c22f61c59d83e40dbf7b59b8a11364" exitCode=60
Oct 31 00:00:02.967455 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.965836    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" event={"ID":"0732ac8c-6272-46d5-950b-c5d92a2216cc","Type":"ContainerDied","Data":"6fd44e83660ce3b1e6a0dc1570672098f3c22f61c59d83e40dbf7b59b8a11364"}
Oct 31 00:00:02.967455 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.966167    4555 scope.go:122] "RemoveContainer" containerID="6fd44e83660ce3b1e6a0dc1570672098f3c22f61c59d83e40dbf7b59b8a11364"
Oct 31 00:00:02.975327 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.975300    4555 generic.go:356] "Generic (PLEG): container finished" podID="d3cac855-9e22-4621-a04c-884e002ead44" containerID="094796671b2bd701aba87b203e4f5e06f5ba6d3dbf91f85e5af04f3717c6853b" exitCode=60
Oct 31 00:00:02.975488 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.975406    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" event={"ID":"d3cac855-9e22-4621-a04c-884e002ead44","Type":"ContainerDied","Data":"094796671b2bd701aba87b203e4f5e06f5ba6d3dbf91f85e5af04f3717c6853b"}
Oct 31 00:00:02.975696 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.975675    4555 scope.go:122] "RemoveContainer" containerID="094796671b2bd701aba87b203e4f5e06f5ba6d3dbf91f85e5af04f3717c6853b"
Oct 31 00:00:02.978604 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.978311    4555 scope.go:122] "RemoveContainer" containerID="9545d42611e3d4c12689b8d357fbb14ed2387015f2ac1099adf0964c288f385e"
Oct 31 00:00:02.978604 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.978497    4555 generic.go:356] "Generic (PLEG): container finished" podID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerID="9545d42611e3d4c12689b8d357fbb14ed2387015f2ac1099adf0964c288f385e" exitCode=60
Oct 31 00:00:02.978604 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:02.978581    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" event={"ID":"0618ce43-2b19-49bb-9dd5-6813e8755b84","Type":"ContainerDied","Data":"9545d42611e3d4c12689b8d357fbb14ed2387015f2ac1099adf0964c288f385e"}
Oct 31 00:00:03.671199 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:03.671116    4555 scope.go:122] "RemoveContainer" containerID="7c168374d46d34653ef98ca3fd60fc14dfe3eb2207e8d2b6c6aaafee39407e17"
Oct 31 00:00:03.992248 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:03.992209    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" event={"ID":"d3cac855-9e22-4621-a04c-884e002ead44","Type":"ContainerStarted","Data":"d710f45c5e0d05fbe84224e5e3ac41535fdb4e2908a496f96198ad313a23151b"}
Oct 31 00:00:03.994922 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:03.994890    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" event={"ID":"0618ce43-2b19-49bb-9dd5-6813e8755b84","Type":"ContainerStarted","Data":"b36389590c21b07cf33ad066213a5fcc67490fb3f8d65c1daf7ebb73e7dbab6f"}
Oct 31 00:00:03.997177 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:03.997146    4555 generic.go:356] "Generic (PLEG): container finished" podID="727002c6-942f-4ea0-b31e-74acc34261d6" containerID="0481ecfea99783d23ac3ede458fe3e984d97e5afa676599ddceb1d5d669c6dae" exitCode=60
Oct 31 00:00:03.997319 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:03.997215    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q" event={"ID":"727002c6-942f-4ea0-b31e-74acc34261d6","Type":"ContainerDied","Data":"0481ecfea99783d23ac3ede458fe3e984d97e5afa676599ddceb1d5d669c6dae"}
Oct 31 00:00:03.997319 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:03.997286    4555 scope.go:122] "RemoveContainer" containerID="4aae15135ae57fe6da01e0747d2328551a4d1c64debbf75c95108656306e51e8"
Oct 31 00:00:03.997581 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:03.997566    4555 scope.go:122] "RemoveContainer" containerID="0481ecfea99783d23ac3ede458fe3e984d97e5afa676599ddceb1d5d669c6dae"
Oct 31 00:00:03.997943 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:03.997917    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q_odh-ai-gateway-infra(727002c6-942f-4ea0-b31e-74acc34261d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q" podUID="727002c6-942f-4ea0-b31e-74acc34261d6"
Oct 31 00:00:03.999813 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:03.999792    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" event={"ID":"0732ac8c-6272-46d5-950b-c5d92a2216cc","Type":"ContainerStarted","Data":"d70387c9befa0207e427bb0f12149c199d5fb1b5b28e735ed80f4c2118d0d84b"}
Oct 31 00:00:04.384437 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:04.384391    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"]
Oct 31 00:00:04.384677 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:04.384646    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" containerID="cri-o://0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da" gracePeriod=30
Oct 31 00:00:05.015037 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.014995    4555 generic.go:356] "Generic (PLEG): container finished" podID="d3cac855-9e22-4621-a04c-884e002ead44" containerID="d710f45c5e0d05fbe84224e5e3ac41535fdb4e2908a496f96198ad313a23151b" exitCode=60
Oct 31 00:00:05.015576 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.015088    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" event={"ID":"d3cac855-9e22-4621-a04c-884e002ead44","Type":"ContainerDied","Data":"d710f45c5e0d05fbe84224e5e3ac41535fdb4e2908a496f96198ad313a23151b"}
Oct 31 00:00:05.015576 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.015173    4555 scope.go:122] "RemoveContainer" containerID="094796671b2bd701aba87b203e4f5e06f5ba6d3dbf91f85e5af04f3717c6853b"
Oct 31 00:00:05.015576 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.015489    4555 scope.go:122] "RemoveContainer" containerID="d710f45c5e0d05fbe84224e5e3ac41535fdb4e2908a496f96198ad313a23151b"
Oct 31 00:00:05.015859 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:05.015828    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j_odh-ai-gateway-infra(d3cac855-9e22-4621-a04c-884e002ead44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" podUID="d3cac855-9e22-4621-a04c-884e002ead44"
Oct 31 00:00:05.034830 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.033419    4555 generic.go:356] "Generic (PLEG): container finished" podID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerID="b36389590c21b07cf33ad066213a5fcc67490fb3f8d65c1daf7ebb73e7dbab6f" exitCode=60
Oct 31 00:00:05.036394 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.036350    4555 scope.go:122] "RemoveContainer" containerID="b36389590c21b07cf33ad066213a5fcc67490fb3f8d65c1daf7ebb73e7dbab6f"
Oct 31 00:00:05.037630 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.036919    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" event={"ID":"0618ce43-2b19-49bb-9dd5-6813e8755b84","Type":"ContainerDied","Data":"b36389590c21b07cf33ad066213a5fcc67490fb3f8d65c1daf7ebb73e7dbab6f"}
Oct 31 00:00:05.039596 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:05.039571    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc_odh-ai-gateway-infra(0618ce43-2b19-49bb-9dd5-6813e8755b84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84"
Oct 31 00:00:05.048137 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.047805    4555 scope.go:122] "RemoveContainer" containerID="0481ecfea99783d23ac3ede458fe3e984d97e5afa676599ddceb1d5d669c6dae"
Oct 31 00:00:05.048258 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:05.048222    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q_odh-ai-gateway-infra(727002c6-942f-4ea0-b31e-74acc34261d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q" podUID="727002c6-942f-4ea0-b31e-74acc34261d6"
Oct 31 00:00:05.059932 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.059882    4555 generic.go:356] "Generic (PLEG): container finished" podID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerID="d70387c9befa0207e427bb0f12149c199d5fb1b5b28e735ed80f4c2118d0d84b" exitCode=60
Oct 31 00:00:05.060072 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.060012    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" event={"ID":"0732ac8c-6272-46d5-950b-c5d92a2216cc","Type":"ContainerDied","Data":"d70387c9befa0207e427bb0f12149c199d5fb1b5b28e735ed80f4c2118d0d84b"}
Oct 31 00:00:05.060512 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.060491    4555 scope.go:122] "RemoveContainer" containerID="d70387c9befa0207e427bb0f12149c199d5fb1b5b28e735ed80f4c2118d0d84b"
Oct 31 00:00:05.060877 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:05.060852    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55_odh-ai-gateway-infra(0732ac8c-6272-46d5-950b-c5d92a2216cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc"
Oct 31 00:00:05.125329 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.125302    4555 scope.go:122] "RemoveContainer" containerID="9545d42611e3d4c12689b8d357fbb14ed2387015f2ac1099adf0964c288f385e"
Oct 31 00:00:05.129529 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.129501    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"]
Oct 31 00:00:05.130075 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.129750    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy" containerID="cri-o://84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78" gracePeriod=30
Oct 31 00:00:05.148704 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:05.148685    4555 scope.go:122] "RemoveContainer" containerID="6fd44e83660ce3b1e6a0dc1570672098f3c22f61c59d83e40dbf7b59b8a11364"
Oct 31 00:00:06.073628 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:06.073600    4555 scope.go:122] "RemoveContainer" containerID="d710f45c5e0d05fbe84224e5e3ac41535fdb4e2908a496f96198ad313a23151b"
Oct 31 00:00:06.074135 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:06.073992    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j_odh-ai-gateway-infra(d3cac855-9e22-4621-a04c-884e002ead44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" podUID="d3cac855-9e22-4621-a04c-884e002ead44"
Oct 31 00:00:06.076611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:06.076585    4555 scope.go:122] "RemoveContainer" containerID="b36389590c21b07cf33ad066213a5fcc67490fb3f8d65c1daf7ebb73e7dbab6f"
Oct 31 00:00:06.077275 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:06.076955    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc_odh-ai-gateway-infra(0618ce43-2b19-49bb-9dd5-6813e8755b84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84"
Oct 31 00:00:06.078536 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:06.078513    4555 scope.go:122] "RemoveContainer" containerID="d70387c9befa0207e427bb0f12149c199d5fb1b5b28e735ed80f4c2118d0d84b"
Oct 31 00:00:06.078840 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:06.078819    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55_odh-ai-gateway-infra(0732ac8c-6272-46d5-950b-c5d92a2216cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc"
Oct 31 00:00:09.275561 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.275522    4555 patch_prober.go:29] interesting pod/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:00:09.276159 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.276126    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:00:09.665629 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.663787    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"]
Oct 31 00:00:09.679334 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.679299    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"]
Oct 31 00:00:09.743799 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.743770    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 31 00:00:09.795439 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.795409    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-kube-api-access-gj94z\" (UniqueName: \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-kube-api-access-gj94z\") pod \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") "
Oct 31 00:00:09.795629 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.795503    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istiod-ca-cert\") pod \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") "
Oct 31 00:00:09.795629 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.795550    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-credential-socket\") pod \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") "
Oct 31 00:00:09.795754 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.795677    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-envoy\") pod \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") "
Oct 31 00:00:09.795817 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.795756    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-token\" (UniqueName: \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-token\") pod \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") "
Oct 31 00:00:09.795876 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.795817    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-data\") pod \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") "
Oct 31 00:00:09.795876 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.795857    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-podinfo\") pod \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") "
Oct 31 00:00:09.795984 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.795901    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-certs\") pod \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") "
Oct 31 00:00:09.795984 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.795943    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-socket\") pod \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\" (UID: \"c907bcbf-31c9-472f-97c2-58b87c67e7f1\") "
Oct 31 00:00:09.795984 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.795944    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istiod-ca-cert" pod "c907bcbf-31c9-472f-97c2-58b87c67e7f1" (UID: "c907bcbf-31c9-472f-97c2-58b87c67e7f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:09.796147 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.796070    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-data" pod "c907bcbf-31c9-472f-97c2-58b87c67e7f1" (UID: "c907bcbf-31c9-472f-97c2-58b87c67e7f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:09.796205 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.796181    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-socket" pod "c907bcbf-31c9-472f-97c2-58b87c67e7f1" (UID: "c907bcbf-31c9-472f-97c2-58b87c67e7f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:09.796399 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.796333    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-certs" pod "c907bcbf-31c9-472f-97c2-58b87c67e7f1" (UID: "c907bcbf-31c9-472f-97c2-58b87c67e7f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:09.797540 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.796864    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-credential-socket" pod "c907bcbf-31c9-472f-97c2-58b87c67e7f1" (UID: "c907bcbf-31c9-472f-97c2-58b87c67e7f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:09.797540 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.797428    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:09.797540 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.797450    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:09.797540 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.797463    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:09.797540 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.797478    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:09.797540 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.797491    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:09.798892 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.798865    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-envoy" pod "c907bcbf-31c9-472f-97c2-58b87c67e7f1" (UID: "c907bcbf-31c9-472f-97c2-58b87c67e7f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:09.799006 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.798985    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-podinfo" pod "c907bcbf-31c9-472f-97c2-58b87c67e7f1" (UID: "c907bcbf-31c9-472f-97c2-58b87c67e7f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:00:09.799083 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.799015    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-kube-api-access-gj94z" pod "c907bcbf-31c9-472f-97c2-58b87c67e7f1" (UID: "c907bcbf-31c9-472f-97c2-58b87c67e7f1"). InnerVolumeSpecName "kube-api-access-gj94z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:09.799535 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.799508    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-token" pod "c907bcbf-31c9-472f-97c2-58b87c67e7f1" (UID: "c907bcbf-31c9-472f-97c2-58b87c67e7f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:09.898884 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.898798    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:09.898884 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.898840    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:09.898884 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.898858    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c907bcbf-31c9-472f-97c2-58b87c67e7f1-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:09.898884 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:09.898872    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj94z\" (UniqueName: \"kubernetes.io/projected/c907bcbf-31c9-472f-97c2-58b87c67e7f1-kube-api-access-gj94z\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:10.123719 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.123356    4555 generic.go:356] "Generic (PLEG): container finished" podID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerID="0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da" exitCode=0
Oct 31 00:00:10.125077 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.124930    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" event={"ID":"c907bcbf-31c9-472f-97c2-58b87c67e7f1","Type":"ContainerDied","Data":"0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da"}
Oct 31 00:00:10.125413 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.125279    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r" event={"ID":"c907bcbf-31c9-472f-97c2-58b87c67e7f1","Type":"ContainerDied","Data":"fa7c9a1c6c39649eb24fe81632796d2903b3ffecc5aad7cccee6aa9443cfc3ea"}
Oct 31 00:00:10.125664 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.125599    4555 scope.go:122] "RemoveContainer" containerID="0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da"
Oct 31 00:00:10.126082 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.126029    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"
Oct 31 00:00:10.184137 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.184113    4555 scope.go:122] "RemoveContainer" containerID="0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da"
Oct 31 00:00:10.184666 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:10.184636    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da\": container with ID starting with 0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da not found: ID does not exist" containerID="0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da"
Oct 31 00:00:10.184850 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.184684    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da"} err="failed to get container status \"0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da\": rpc error: code = NotFound desc = could not find container \"0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da\": container with ID starting with 0a511394a05e4aa3b77d2b25e990dda203aa90b7706137216e4be69e982381da not found: ID does not exist"
Oct 31 00:00:10.188804 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.188776    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"]
Oct 31 00:00:10.192256 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.192234    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e15304-openshift-default-bc69747c7-qqk5r"]
Oct 31 00:00:10.409281 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.409255    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 31 00:00:10.506973 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.506941    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-credential-socket\") pod \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") "
Oct 31 00:00:10.507154 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.506986    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-token\" (UniqueName: \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-token\") pod \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") "
Oct 31 00:00:10.507154 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507025    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-envoy\") pod \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") "
Oct 31 00:00:10.507154 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507070    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-data\") pod \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") "
Oct 31 00:00:10.507292 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507192    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-kube-api-access-x6flf\" (UniqueName: \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-kube-api-access-x6flf\") pod \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") "
Oct 31 00:00:10.507292 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507250    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-socket\") pod \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") "
Oct 31 00:00:10.507430 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507290    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istiod-ca-cert\") pod \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") "
Oct 31 00:00:10.507430 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507425    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-certs\") pod \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") "
Oct 31 00:00:10.507544 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507459    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-podinfo\") pod \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\" (UID: \"c2029c09-8186-46dd-bc4f-aab88bde8fd1\") "
Oct 31 00:00:10.507544 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507282    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-credential-socket" pod "c2029c09-8186-46dd-bc4f-aab88bde8fd1" (UID: "c2029c09-8186-46dd-bc4f-aab88bde8fd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:10.507544 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507512    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-socket" pod "c2029c09-8186-46dd-bc4f-aab88bde8fd1" (UID: "c2029c09-8186-46dd-bc4f-aab88bde8fd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:10.507544 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507488    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-data" pod "c2029c09-8186-46dd-bc4f-aab88bde8fd1" (UID: "c2029c09-8186-46dd-bc4f-aab88bde8fd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:10.507792 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507724    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-certs" pod "c2029c09-8186-46dd-bc4f-aab88bde8fd1" (UID: "c2029c09-8186-46dd-bc4f-aab88bde8fd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:10.507792 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.507765    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istiod-ca-cert" pod "c2029c09-8186-46dd-bc4f-aab88bde8fd1" (UID: "c2029c09-8186-46dd-bc4f-aab88bde8fd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:10.508069 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.508048    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:10.508178 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.508072    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:10.508178 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.508088    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:10.508178 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.508100    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:10.508178 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.508115    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:10.509448 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.509421    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-token" pod "c2029c09-8186-46dd-bc4f-aab88bde8fd1" (UID: "c2029c09-8186-46dd-bc4f-aab88bde8fd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:10.509562 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.509544    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-kube-api-access-x6flf" pod "c2029c09-8186-46dd-bc4f-aab88bde8fd1" (UID: "c2029c09-8186-46dd-bc4f-aab88bde8fd1"). InnerVolumeSpecName "kube-api-access-x6flf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:10.509945 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.509922    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-envoy" pod "c2029c09-8186-46dd-bc4f-aab88bde8fd1" (UID: "c2029c09-8186-46dd-bc4f-aab88bde8fd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:10.510019 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.509998    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-podinfo" pod "c2029c09-8186-46dd-bc4f-aab88bde8fd1" (UID: "c2029c09-8186-46dd-bc4f-aab88bde8fd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:00:10.609158 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.609124    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6flf\" (UniqueName: \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-kube-api-access-x6flf\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:10.609158 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.609153    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:10.609158 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.609163    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:10.609425 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.609172    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2029c09-8186-46dd-bc4f-aab88bde8fd1-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:10.724146 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:10.724065    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"]
Oct 31 00:00:11.139217 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.139180    4555 generic.go:356] "Generic (PLEG): container finished" podID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerID="84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78" exitCode=0
Oct 31 00:00:11.139396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.139238    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"
Oct 31 00:00:11.139396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.139256    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" event={"ID":"c2029c09-8186-46dd-bc4f-aab88bde8fd1","Type":"ContainerDied","Data":"84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78"}
Oct 31 00:00:11.139396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.139320    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8" event={"ID":"c2029c09-8186-46dd-bc4f-aab88bde8fd1","Type":"ContainerDied","Data":"1b66c5692e2596260761f33a4d0ba1be067ae16689f064dfaa7e3bf805857d2c"}
Oct 31 00:00:11.139396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.139344    4555 scope.go:122] "RemoveContainer" containerID="84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78"
Oct 31 00:00:11.151558 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.151540    4555 scope.go:122] "RemoveContainer" containerID="3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3"
Oct 31 00:00:11.166218 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.166159    4555 scope.go:122] "RemoveContainer" containerID="84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78"
Oct 31 00:00:11.166642 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:11.166614    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78\": container with ID starting with 84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78 not found: ID does not exist" containerID="84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78"
Oct 31 00:00:11.166772 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.166655    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78"} err="failed to get container status \"84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78\": rpc error: code = NotFound desc = could not find container \"84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78\": container with ID starting with 84c956bec08825dbde7869473f84a42cb14f822ab69e5a914c610169461a3d78 not found: ID does not exist"
Oct 31 00:00:11.166772 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.166682    4555 scope.go:122] "RemoveContainer" containerID="3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3"
Oct 31 00:00:11.167077 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:11.167042    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3\": container with ID starting with 3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3 not found: ID does not exist" containerID="3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3"
Oct 31 00:00:11.167146 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.167086    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3"} err="failed to get container status \"3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3\": rpc error: code = NotFound desc = could not find container \"3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3\": container with ID starting with 3cb43359ebdffa8a89e75c049f17023d99c3df9deda0f22ccbbbb371c3aba9f3 not found: ID does not exist"
Oct 31 00:00:11.174400 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.173419    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"]
Oct 31 00:00:11.189431 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.189392    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a3e876-openshift-default-68698799f-xd2x8"]
Oct 31 00:00:11.210266 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.210233    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" path="/var/lib/kubelet/pods/c2029c09-8186-46dd-bc4f-aab88bde8fd1/volumes"
Oct 31 00:00:11.210829 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:11.210813    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" path="/var/lib/kubelet/pods/c907bcbf-31c9-472f-97c2-58b87c67e7f1/volumes"
Oct 31 00:00:14.094727 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.094671    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" containerID="cri-o://fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a" gracePeriod=45
Oct 31 00:00:14.104711 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.103761    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": EOF"
Oct 31 00:00:14.238415 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.238362    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"]
Oct 31 00:00:14.238711 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.238658    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl" podUID="a1c131d4-4156-40bf-9f1e-e62079bd51a8" containerName="maas-api" containerID="cri-o://df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772" gracePeriod=30
Oct 31 00:00:14.665395 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.665147    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4" podUID="2918e4a9-cdfb-460f-82f5-941944ce7a47" containerName="payload-processing" containerID="cri-o://682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74" gracePeriod=25
Oct 31 00:00:14.669808 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.669768    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 31 00:00:14.680243 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.680189    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9" podUID="b3cd0b30-12b8-4c3a-b1ca-dba8525e7190" containerName="payload-pre-processing" containerID="cri-o://28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa" gracePeriod=25
Oct 31 00:00:14.774696 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.774663    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-tmp-dir\") pod \"56c6ddb0-5235-4458-a035-b94f0588a3bf\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") "
Oct 31 00:00:14.775336 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.775315    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-dshm\") pod \"56c6ddb0-5235-4458-a035-b94f0588a3bf\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") "
Oct 31 00:00:14.775979 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.775960    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-model-cache\") pod \"56c6ddb0-5235-4458-a035-b94f0588a3bf\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") "
Oct 31 00:00:14.776505 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.775166    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-tmp-dir" pod "56c6ddb0-5235-4458-a035-b94f0588a3bf" (UID: "56c6ddb0-5235-4458-a035-b94f0588a3bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:14.777133 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.776309    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-model-cache" pod "56c6ddb0-5235-4458-a035-b94f0588a3bf" (UID: "56c6ddb0-5235-4458-a035-b94f0588a3bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:14.777275 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.776436    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56c6ddb0-5235-4458-a035-b94f0588a3bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/56c6ddb0-5235-4458-a035-b94f0588a3bf-tls-certs\") pod \"56c6ddb0-5235-4458-a035-b94f0588a3bf\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") "
Oct 31 00:00:14.777464 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.777449    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-home\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-home\") pod \"56c6ddb0-5235-4458-a035-b94f0588a3bf\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") "
Oct 31 00:00:14.777921 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.777795    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-home" pod "56c6ddb0-5235-4458-a035-b94f0588a3bf" (UID: "56c6ddb0-5235-4458-a035-b94f0588a3bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:14.778206 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.778156    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c6ddb0-5235-4458-a035-b94f0588a3bf-kube-api-access-p4sjw\" (UniqueName: \"kubernetes.io/projected/56c6ddb0-5235-4458-a035-b94f0588a3bf-kube-api-access-p4sjw\") pod \"56c6ddb0-5235-4458-a035-b94f0588a3bf\" (UID: \"56c6ddb0-5235-4458-a035-b94f0588a3bf\") "
Oct 31 00:00:14.779292 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.779270    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:14.779490 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.779477    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:14.779599 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.779586    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:14.784619 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.783657    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c6ddb0-5235-4458-a035-b94f0588a3bf-kube-api-access-p4sjw" pod "56c6ddb0-5235-4458-a035-b94f0588a3bf" (UID: "56c6ddb0-5235-4458-a035-b94f0588a3bf"). InnerVolumeSpecName "kube-api-access-p4sjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:14.784901 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.784864    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-dshm" pod "56c6ddb0-5235-4458-a035-b94f0588a3bf" (UID: "56c6ddb0-5235-4458-a035-b94f0588a3bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:14.787116 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.787087    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c6ddb0-5235-4458-a035-b94f0588a3bf-tls-certs" pod "56c6ddb0-5235-4458-a035-b94f0588a3bf" (UID: "56c6ddb0-5235-4458-a035-b94f0588a3bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:00:14.844460 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.844065    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 31 00:00:14.898606 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.897324    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c131d4-4156-40bf-9f1e-e62079bd51a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c131d4-4156-40bf-9f1e-e62079bd51a8-openshift-service-ca\") pod \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") "
Oct 31 00:00:14.898606 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.897401    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-maas-api-tls\") pod \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") "
Oct 31 00:00:14.898606 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.897573    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c131d4-4156-40bf-9f1e-e62079bd51a8-kube-api-access-8rp7x\" (UniqueName: \"kubernetes.io/projected/a1c131d4-4156-40bf-9f1e-e62079bd51a8-kube-api-access-8rp7x\") pod \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") "
Oct 31 00:00:14.898606 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.897687    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-metrics-certs\") pod \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\" (UID: \"a1c131d4-4156-40bf-9f1e-e62079bd51a8\") "
Oct 31 00:00:14.898606 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.898107    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c6ddb0-5235-4458-a035-b94f0588a3bf-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:14.898606 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.898126    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4sjw\" (UniqueName: \"kubernetes.io/projected/56c6ddb0-5235-4458-a035-b94f0588a3bf-kube-api-access-p4sjw\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:14.898606 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.898141    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c6ddb0-5235-4458-a035-b94f0588a3bf-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:14.901721 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.901676    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c131d4-4156-40bf-9f1e-e62079bd51a8-openshift-service-ca" pod "a1c131d4-4156-40bf-9f1e-e62079bd51a8" (UID: "a1c131d4-4156-40bf-9f1e-e62079bd51a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:14.903981 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.903952    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c131d4-4156-40bf-9f1e-e62079bd51a8-kube-api-access-8rp7x" pod "a1c131d4-4156-40bf-9f1e-e62079bd51a8" (UID: "a1c131d4-4156-40bf-9f1e-e62079bd51a8"). InnerVolumeSpecName "kube-api-access-8rp7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:14.925026 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.924507    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-maas-api-tls" pod "a1c131d4-4156-40bf-9f1e-e62079bd51a8" (UID: "a1c131d4-4156-40bf-9f1e-e62079bd51a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:00:14.925219 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.925073    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-metrics-certs" pod "a1c131d4-4156-40bf-9f1e-e62079bd51a8" (UID: "a1c131d4-4156-40bf-9f1e-e62079bd51a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:00:14.999665 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.999569    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rp7x\" (UniqueName: \"kubernetes.io/projected/a1c131d4-4156-40bf-9f1e-e62079bd51a8-kube-api-access-8rp7x\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:14.999665 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.999606    4555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-metrics-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:14.999665 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.999623    4555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c131d4-4156-40bf-9f1e-e62079bd51a8-openshift-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:14.999665 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:14.999638    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c131d4-4156-40bf-9f1e-e62079bd51a8-maas-api-tls\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:15.030518 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.030481    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 31 00:00:15.077246 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.076933    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 31 00:00:15.100558 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.100522    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2918e4a9-cdfb-460f-82f5-941944ce7a47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2918e4a9-cdfb-460f-82f5-941944ce7a47-plugins-config-volume\") pod \"2918e4a9-cdfb-460f-82f5-941944ce7a47\" (UID: \"2918e4a9-cdfb-460f-82f5-941944ce7a47\") "
Oct 31 00:00:15.101123 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.100973    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2918e4a9-cdfb-460f-82f5-941944ce7a47-kube-api-access-6dpl4\" (UniqueName: \"kubernetes.io/projected/2918e4a9-cdfb-460f-82f5-941944ce7a47-kube-api-access-6dpl4\") pod \"2918e4a9-cdfb-460f-82f5-941944ce7a47\" (UID: \"2918e4a9-cdfb-460f-82f5-941944ce7a47\") "
Oct 31 00:00:15.105651 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.105621    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2918e4a9-cdfb-460f-82f5-941944ce7a47-kube-api-access-6dpl4" pod "2918e4a9-cdfb-460f-82f5-941944ce7a47" (UID: "2918e4a9-cdfb-460f-82f5-941944ce7a47"). InnerVolumeSpecName "kube-api-access-6dpl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:15.106678 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.106648    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2918e4a9-cdfb-460f-82f5-941944ce7a47-plugins-config-volume" pod "2918e4a9-cdfb-460f-82f5-941944ce7a47" (UID: "2918e4a9-cdfb-460f-82f5-941944ce7a47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:15.175337 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.175225    4555 generic.go:356] "Generic (PLEG): container finished" podID="2918e4a9-cdfb-460f-82f5-941944ce7a47" containerID="682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74" exitCode=0
Oct 31 00:00:15.175337 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.175304    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"
Oct 31 00:00:15.175594 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.175307    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4" event={"ID":"2918e4a9-cdfb-460f-82f5-941944ce7a47","Type":"ContainerDied","Data":"682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74"}
Oct 31 00:00:15.175594 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.175457    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4" event={"ID":"2918e4a9-cdfb-460f-82f5-941944ce7a47","Type":"ContainerDied","Data":"acafffbc37c533243bcfe1d36a848d39917f122e8df8b5e84797ae46d54e3ed9"}
Oct 31 00:00:15.175594 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.175485    4555 scope.go:122] "RemoveContainer" containerID="682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74"
Oct 31 00:00:15.179687 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.179505    4555 generic.go:356] "Generic (PLEG): container finished" podID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerID="fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a" exitCode=0
Oct 31 00:00:15.179687 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.179625    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" event={"ID":"56c6ddb0-5235-4458-a035-b94f0588a3bf","Type":"ContainerDied","Data":"fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a"}
Oct 31 00:00:15.179687 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.179652    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk" event={"ID":"56c6ddb0-5235-4458-a035-b94f0588a3bf","Type":"ContainerDied","Data":"7871f9fc86d72cd4b591686cd14d55878e0fa7a63a2cb17ed97a70781f5565df"}
Oct 31 00:00:15.179878 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.179712    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"
Oct 31 00:00:15.187143 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.187097    4555 generic.go:356] "Generic (PLEG): container finished" podID="b3cd0b30-12b8-4c3a-b1ca-dba8525e7190" containerID="28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa" exitCode=0
Oct 31 00:00:15.187298 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.187209    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"
Oct 31 00:00:15.187298 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.187258    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9" event={"ID":"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190","Type":"ContainerDied","Data":"28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa"}
Oct 31 00:00:15.187484 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.187334    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9" event={"ID":"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190","Type":"ContainerDied","Data":"9c7a07b364dd8daf238041160f34a537a7d8d1a823fae4fa1430048bc05e9426"}
Oct 31 00:00:15.191111 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.191088    4555 generic.go:356] "Generic (PLEG): container finished" podID="a1c131d4-4156-40bf-9f1e-e62079bd51a8" containerID="df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772" exitCode=0
Oct 31 00:00:15.191288 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.191260    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl" event={"ID":"a1c131d4-4156-40bf-9f1e-e62079bd51a8","Type":"ContainerDied","Data":"df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772"}
Oct 31 00:00:15.191494 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.191477    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl" event={"ID":"a1c131d4-4156-40bf-9f1e-e62079bd51a8","Type":"ContainerDied","Data":"8a9226c36097c245a7953701404cffadbe67876cf7fdeb814479963559b81e93"}
Oct 31 00:00:15.191658 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.191299    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"
Oct 31 00:00:15.200330 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.200302    4555 scope.go:122] "RemoveContainer" containerID="b36389590c21b07cf33ad066213a5fcc67490fb3f8d65c1daf7ebb73e7dbab6f"
Oct 31 00:00:15.200648 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.200628    4555 scope.go:122] "RemoveContainer" containerID="d70387c9befa0207e427bb0f12149c199d5fb1b5b28e735ed80f4c2118d0d84b"
Oct 31 00:00:15.201254 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.201234    4555 scope.go:122] "RemoveContainer" containerID="0481ecfea99783d23ac3ede458fe3e984d97e5afa676599ddceb1d5d669c6dae"
Oct 31 00:00:15.201842 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.201819    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-kube-api-access-4fpwv\" (UniqueName: \"kubernetes.io/projected/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-kube-api-access-4fpwv\") pod \"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190\" (UID: \"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190\") "
Oct 31 00:00:15.202027 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.202007    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-plugins-config-volume\") pod \"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190\" (UID: \"b3cd0b30-12b8-4c3a-b1ca-dba8525e7190\") "
Oct 31 00:00:15.202362 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.202342    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dpl4\" (UniqueName: \"kubernetes.io/projected/2918e4a9-cdfb-460f-82f5-941944ce7a47-kube-api-access-6dpl4\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:15.202467 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.202400    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2918e4a9-cdfb-460f-82f5-941944ce7a47-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:15.202673 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.202632    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-plugins-config-volume" pod "b3cd0b30-12b8-4c3a-b1ca-dba8525e7190" (UID: "b3cd0b30-12b8-4c3a-b1ca-dba8525e7190"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:15.204168 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.204150    4555 scope.go:122] "RemoveContainer" containerID="682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74"
Oct 31 00:00:15.204839 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:15.204806    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74\": container with ID starting with 682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74 not found: ID does not exist" containerID="682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74"
Oct 31 00:00:15.204937 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.204849    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74"} err="failed to get container status \"682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74\": rpc error: code = NotFound desc = could not find container \"682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74\": container with ID starting with 682933c30df3874a021a5ddf887adf6be194d553b3cdf47d39e708c1acff1e74 not found: ID does not exist"
Oct 31 00:00:15.204937 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.204871    4555 scope.go:122] "RemoveContainer" containerID="fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a"
Oct 31 00:00:15.212682 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.211825    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-kube-api-access-4fpwv" pod "b3cd0b30-12b8-4c3a-b1ca-dba8525e7190" (UID: "b3cd0b30-12b8-4c3a-b1ca-dba8525e7190"). InnerVolumeSpecName "kube-api-access-4fpwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:15.241587 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.241565    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"]
Oct 31 00:00:15.241677 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.241575    4555 scope.go:122] "RemoveContainer" containerID="fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a"
Oct 31 00:00:15.241984 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:15.241946    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a\": container with ID starting with fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a not found: ID does not exist" containerID="fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a"
Oct 31 00:00:15.242038 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.241987    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a"} err="failed to get container status \"fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a\": rpc error: code = NotFound desc = could not find container \"fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a\": container with ID starting with fe9459654d8e35596c222f119229b8405187f167acea6878e983c1822352319a not found: ID does not exist"
Oct 31 00:00:15.242038 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.242009    4555 scope.go:122] "RemoveContainer" containerID="28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa"
Oct 31 00:00:15.246167 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.246142    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4e6c3b-7ff8b964f5-9fsj4"]
Oct 31 00:00:15.272308 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.272285    4555 scope.go:122] "RemoveContainer" containerID="28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa"
Oct 31 00:00:15.272682 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:15.272658    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa\": container with ID starting with 28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa not found: ID does not exist" containerID="28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa"
Oct 31 00:00:15.272798 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.272689    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa"} err="failed to get container status \"28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa\": rpc error: code = NotFound desc = could not find container \"28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa\": container with ID starting with 28cdf3e21404f176b5da5470f724b6905504ec5ef67b8d5395038a007f68f4fa not found: ID does not exist"
Oct 31 00:00:15.272798 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.272709    4555 scope.go:122] "RemoveContainer" containerID="df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772"
Oct 31 00:00:15.275889 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.275864    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"]
Oct 31 00:00:15.287034 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.287011    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-200408-7ddcb6445b-jkvrl"]
Oct 31 00:00:15.299140 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.294936    4555 scope.go:122] "RemoveContainer" containerID="df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772"
Oct 31 00:00:15.299424 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:15.299318    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772\": container with ID starting with df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772 not found: ID does not exist" containerID="df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772"
Oct 31 00:00:15.299424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.299396    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772"} err="failed to get container status \"df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772\": rpc error: code = NotFound desc = could not find container \"df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772\": container with ID starting with df3890e70d9f94427425856b2f96699b3b70e9339458a58b673e2f06e6c93772 not found: ID does not exist"
Oct 31 00:00:15.303710 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.303683    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:15.303804 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.303716    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpwv\" (UniqueName: \"kubernetes.io/projected/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190-kube-api-access-4fpwv\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:15.309820 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.309796    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"]
Oct 31 00:00:15.320530 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.320504    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-843d8c/simulator-w4-843d8c-kserve-6655dd6b4f-9s7wk"]
Oct 31 00:00:15.512533 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.512500    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"]
Oct 31 00:00:15.515858 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:15.515837    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4e6c3b-646cc5d67f-5k7d9"]
Oct 31 00:00:16.121788 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:16.121752    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"]
Oct 31 00:00:16.204576 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:16.204540    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" event={"ID":"0732ac8c-6272-46d5-950b-c5d92a2216cc","Type":"ContainerStarted","Data":"b8db4471975f514d9192fafe4f753ac49c9128a2058e678e9b0726b2ed6c1ed2"}
Oct 31 00:00:16.208273 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:16.208236    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" event={"ID":"0618ce43-2b19-49bb-9dd5-6813e8755b84","Type":"ContainerStarted","Data":"54b0d1a5015072a1c8fbb148123945e3a478b236c20acf003755be9cd347fc7a"}
Oct 31 00:00:16.212159 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:16.212129    4555 generic.go:356] "Generic (PLEG): container finished" podID="727002c6-942f-4ea0-b31e-74acc34261d6" containerID="9ba4e9c97f85bfa7ddc4da8475b2176d531991c9b59d07cb75cc396ca0e5799f" exitCode=60
Oct 31 00:00:16.212284 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:16.212212    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q" event={"ID":"727002c6-942f-4ea0-b31e-74acc34261d6","Type":"ContainerDied","Data":"9ba4e9c97f85bfa7ddc4da8475b2176d531991c9b59d07cb75cc396ca0e5799f"}
Oct 31 00:00:16.212284 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:16.212258    4555 scope.go:122] "RemoveContainer" containerID="0481ecfea99783d23ac3ede458fe3e984d97e5afa676599ddceb1d5d669c6dae"
Oct 31 00:00:16.212680 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:16.212664    4555 scope.go:122] "RemoveContainer" containerID="9ba4e9c97f85bfa7ddc4da8475b2176d531991c9b59d07cb75cc396ca0e5799f"
Oct 31 00:00:16.213006 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:16.212982    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q_odh-ai-gateway-infra(727002c6-942f-4ea0-b31e-74acc34261d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q" podUID="727002c6-942f-4ea0-b31e-74acc34261d6"
Oct 31 00:00:17.210202 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.210154    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2918e4a9-cdfb-460f-82f5-941944ce7a47" path="/var/lib/kubelet/pods/2918e4a9-cdfb-460f-82f5-941944ce7a47/volumes"
Oct 31 00:00:17.210779 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.210669    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" path="/var/lib/kubelet/pods/56c6ddb0-5235-4458-a035-b94f0588a3bf/volumes"
Oct 31 00:00:17.211099 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.211082    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c131d4-4156-40bf-9f1e-e62079bd51a8" path="/var/lib/kubelet/pods/a1c131d4-4156-40bf-9f1e-e62079bd51a8/volumes"
Oct 31 00:00:17.212417 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.211953    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cd0b30-12b8-4c3a-b1ca-dba8525e7190" path="/var/lib/kubelet/pods/b3cd0b30-12b8-4c3a-b1ca-dba8525e7190/volumes"
Oct 31 00:00:17.227703 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.227676    4555 generic.go:356] "Generic (PLEG): container finished" podID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerID="54b0d1a5015072a1c8fbb148123945e3a478b236c20acf003755be9cd347fc7a" exitCode=60
Oct 31 00:00:17.227852 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.227755    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" event={"ID":"0618ce43-2b19-49bb-9dd5-6813e8755b84","Type":"ContainerDied","Data":"54b0d1a5015072a1c8fbb148123945e3a478b236c20acf003755be9cd347fc7a"}
Oct 31 00:00:17.227852 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.227812    4555 scope.go:122] "RemoveContainer" containerID="b36389590c21b07cf33ad066213a5fcc67490fb3f8d65c1daf7ebb73e7dbab6f"
Oct 31 00:00:17.228234 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.228202    4555 scope.go:122] "RemoveContainer" containerID="54b0d1a5015072a1c8fbb148123945e3a478b236c20acf003755be9cd347fc7a"
Oct 31 00:00:17.228549 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:17.228522    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc_odh-ai-gateway-infra(0618ce43-2b19-49bb-9dd5-6813e8755b84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84"
Oct 31 00:00:17.233348 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.233329    4555 generic.go:356] "Generic (PLEG): container finished" podID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerID="b8db4471975f514d9192fafe4f753ac49c9128a2058e678e9b0726b2ed6c1ed2" exitCode=60
Oct 31 00:00:17.233465 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.233420    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" event={"ID":"0732ac8c-6272-46d5-950b-c5d92a2216cc","Type":"ContainerDied","Data":"b8db4471975f514d9192fafe4f753ac49c9128a2058e678e9b0726b2ed6c1ed2"}
Oct 31 00:00:17.233805 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.233786    4555 scope.go:122] "RemoveContainer" containerID="b8db4471975f514d9192fafe4f753ac49c9128a2058e678e9b0726b2ed6c1ed2"
Oct 31 00:00:17.234083 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:17.234059    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55_odh-ai-gateway-infra(0732ac8c-6272-46d5-950b-c5d92a2216cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc"
Oct 31 00:00:17.243184 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.243166    4555 scope.go:122] "RemoveContainer" containerID="d70387c9befa0207e427bb0f12149c199d5fb1b5b28e735ed80f4c2118d0d84b"
Oct 31 00:00:17.260642 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.260618    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"]
Oct 31 00:00:17.309396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.309016    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"]
Oct 31 00:00:17.314163 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.314135    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"]
Oct 31 00:00:17.539391 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.533839    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"
Oct 31 00:00:17.635756 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.635722    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727002c6-942f-4ea0-b31e-74acc34261d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727002c6-942f-4ea0-b31e-74acc34261d6-maas-api-service-ca\") pod \"727002c6-942f-4ea0-b31e-74acc34261d6\" (UID: \"727002c6-942f-4ea0-b31e-74acc34261d6\") "
Oct 31 00:00:17.635951 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.635930    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727002c6-942f-4ea0-b31e-74acc34261d6-kube-api-access-j489b\" (UniqueName: \"kubernetes.io/projected/727002c6-942f-4ea0-b31e-74acc34261d6-kube-api-access-j489b\") pod \"727002c6-942f-4ea0-b31e-74acc34261d6\" (UID: \"727002c6-942f-4ea0-b31e-74acc34261d6\") "
Oct 31 00:00:17.636187 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.636157    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727002c6-942f-4ea0-b31e-74acc34261d6-maas-api-service-ca" pod "727002c6-942f-4ea0-b31e-74acc34261d6" (UID: "727002c6-942f-4ea0-b31e-74acc34261d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:17.636497 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.636473    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727002c6-942f-4ea0-b31e-74acc34261d6-maas-api-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:17.638201 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.638176    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727002c6-942f-4ea0-b31e-74acc34261d6-kube-api-access-j489b" pod "727002c6-942f-4ea0-b31e-74acc34261d6" (UID: "727002c6-942f-4ea0-b31e-74acc34261d6"). InnerVolumeSpecName "kube-api-access-j489b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:17.737891 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:17.737850    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j489b\" (UniqueName: \"kubernetes.io/projected/727002c6-942f-4ea0-b31e-74acc34261d6-kube-api-access-j489b\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:18.200779 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.200748    4555 scope.go:122] "RemoveContainer" containerID="d710f45c5e0d05fbe84224e5e3ac41535fdb4e2908a496f96198ad313a23151b"
Oct 31 00:00:18.253583 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.250498    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q" event={"ID":"727002c6-942f-4ea0-b31e-74acc34261d6","Type":"ContainerDied","Data":"4c46fe9150540a23acdc6b57ac92bdfd7c0b50d6951d92b763de5b0b24e581ee"}
Oct 31 00:00:18.253583 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.250574    4555 scope.go:122] "RemoveContainer" containerID="9ba4e9c97f85bfa7ddc4da8475b2176d531991c9b59d07cb75cc396ca0e5799f"
Oct 31 00:00:18.253583 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.251275    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"
Oct 31 00:00:18.465946 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.460865    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"]
Oct 31 00:00:18.475393 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.472840    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843d8c-29890080-z8n6q"]
Oct 31 00:00:18.662719 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.662688    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"
Oct 31 00:00:18.735447 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.732960    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"
Oct 31 00:00:18.766662 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.766628    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0618ce43-2b19-49bb-9dd5-6813e8755b84-kube-api-access-lkx9q\" (UniqueName: \"kubernetes.io/projected/0618ce43-2b19-49bb-9dd5-6813e8755b84-kube-api-access-lkx9q\") pod \"0618ce43-2b19-49bb-9dd5-6813e8755b84\" (UID: \"0618ce43-2b19-49bb-9dd5-6813e8755b84\") "
Oct 31 00:00:18.766835 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.766704    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0618ce43-2b19-49bb-9dd5-6813e8755b84-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0618ce43-2b19-49bb-9dd5-6813e8755b84-maas-api-service-ca\") pod \"0618ce43-2b19-49bb-9dd5-6813e8755b84\" (UID: \"0618ce43-2b19-49bb-9dd5-6813e8755b84\") "
Oct 31 00:00:18.767646 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.767612    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0618ce43-2b19-49bb-9dd5-6813e8755b84-maas-api-service-ca" pod "0618ce43-2b19-49bb-9dd5-6813e8755b84" (UID: "0618ce43-2b19-49bb-9dd5-6813e8755b84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:18.785391 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.771444    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0618ce43-2b19-49bb-9dd5-6813e8755b84-kube-api-access-lkx9q" pod "0618ce43-2b19-49bb-9dd5-6813e8755b84" (UID: "0618ce43-2b19-49bb-9dd5-6813e8755b84"). InnerVolumeSpecName "kube-api-access-lkx9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:18.867952 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.867913    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0732ac8c-6272-46d5-950b-c5d92a2216cc-kube-api-access-qjxwh\" (UniqueName: \"kubernetes.io/projected/0732ac8c-6272-46d5-950b-c5d92a2216cc-kube-api-access-qjxwh\") pod \"0732ac8c-6272-46d5-950b-c5d92a2216cc\" (UID: \"0732ac8c-6272-46d5-950b-c5d92a2216cc\") "
Oct 31 00:00:18.868154 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.868079    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0732ac8c-6272-46d5-950b-c5d92a2216cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0732ac8c-6272-46d5-950b-c5d92a2216cc-maas-api-service-ca\") pod \"0732ac8c-6272-46d5-950b-c5d92a2216cc\" (UID: \"0732ac8c-6272-46d5-950b-c5d92a2216cc\") "
Oct 31 00:00:18.868576 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.868547    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkx9q\" (UniqueName: \"kubernetes.io/projected/0618ce43-2b19-49bb-9dd5-6813e8755b84-kube-api-access-lkx9q\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:18.868707 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.868578    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0618ce43-2b19-49bb-9dd5-6813e8755b84-maas-api-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:18.868956 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.868902    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0732ac8c-6272-46d5-950b-c5d92a2216cc-maas-api-service-ca" pod "0732ac8c-6272-46d5-950b-c5d92a2216cc" (UID: "0732ac8c-6272-46d5-950b-c5d92a2216cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:18.870362 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.870334    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0732ac8c-6272-46d5-950b-c5d92a2216cc-kube-api-access-qjxwh" pod "0732ac8c-6272-46d5-950b-c5d92a2216cc" (UID: "0732ac8c-6272-46d5-950b-c5d92a2216cc"). InnerVolumeSpecName "kube-api-access-qjxwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:18.969882 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.969837    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjxwh\" (UniqueName: \"kubernetes.io/projected/0732ac8c-6272-46d5-950b-c5d92a2216cc-kube-api-access-qjxwh\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:18.969882 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:18.969881    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0732ac8c-6272-46d5-950b-c5d92a2216cc-maas-api-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:19.211179 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.211139    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727002c6-942f-4ea0-b31e-74acc34261d6" path="/var/lib/kubelet/pods/727002c6-942f-4ea0-b31e-74acc34261d6/volumes"
Oct 31 00:00:19.282762 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.282734    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"
Oct 31 00:00:19.283210 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.282772    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55" event={"ID":"0732ac8c-6272-46d5-950b-c5d92a2216cc","Type":"ContainerDied","Data":"b407ade22688d2047858ae0202ed5641f96d2efeaecfcb2f77e3b43f7af366db"}
Oct 31 00:00:19.283210 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.282845    4555 scope.go:122] "RemoveContainer" containerID="b8db4471975f514d9192fafe4f753ac49c9128a2058e678e9b0726b2ed6c1ed2"
Oct 31 00:00:19.285527 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.285501    4555 generic.go:356] "Generic (PLEG): container finished" podID="d3cac855-9e22-4621-a04c-884e002ead44" containerID="03f70898a549a608ad8572cdc43246517af41f0e9e0e19a033e8269e414c987f" exitCode=60
Oct 31 00:00:19.285652 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.285564    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" event={"ID":"d3cac855-9e22-4621-a04c-884e002ead44","Type":"ContainerDied","Data":"03f70898a549a608ad8572cdc43246517af41f0e9e0e19a033e8269e414c987f"}
Oct 31 00:00:19.286034 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.286013    4555 scope.go:122] "RemoveContainer" containerID="03f70898a549a608ad8572cdc43246517af41f0e9e0e19a033e8269e414c987f"
Oct 31 00:00:19.286363 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:19.286338    4555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j_odh-ai-gateway-infra(d3cac855-9e22-4621-a04c-884e002ead44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" podUID="d3cac855-9e22-4621-a04c-884e002ead44"
Oct 31 00:00:19.288711 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.287569    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc" event={"ID":"0618ce43-2b19-49bb-9dd5-6813e8755b84","Type":"ContainerDied","Data":"b6ca65b6e6928b36327d7f9d4045b4438f45bbd78d7ccd9720b559bf3a2dbcc5"}
Oct 31 00:00:19.288711 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.287682    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"
Oct 31 00:00:19.301995 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.301972    4555 scope.go:122] "RemoveContainer" containerID="d710f45c5e0d05fbe84224e5e3ac41535fdb4e2908a496f96198ad313a23151b"
Oct 31 00:00:19.317106 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.317086    4555 scope.go:122] "RemoveContainer" containerID="54b0d1a5015072a1c8fbb148123945e3a478b236c20acf003755be9cd347fc7a"
Oct 31 00:00:19.380338 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.380284    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"]
Oct 31 00:00:19.395157 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.395125    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e5934f-29890080-5sb55"]
Oct 31 00:00:19.409257 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.409227    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"]
Oct 31 00:00:19.419185 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:19.419161    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-86d846-29890080-9skmc"]
Oct 31 00:00:20.332226 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:20.332196    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"]
Oct 31 00:00:20.500696 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:20.500668    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"
Oct 31 00:00:20.589325 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:20.589225    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3cac855-9e22-4621-a04c-884e002ead44-kube-api-access-79fhk\" (UniqueName: \"kubernetes.io/projected/d3cac855-9e22-4621-a04c-884e002ead44-kube-api-access-79fhk\") pod \"d3cac855-9e22-4621-a04c-884e002ead44\" (UID: \"d3cac855-9e22-4621-a04c-884e002ead44\") "
Oct 31 00:00:20.589325 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:20.589297    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3cac855-9e22-4621-a04c-884e002ead44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3cac855-9e22-4621-a04c-884e002ead44-maas-api-service-ca\") pod \"d3cac855-9e22-4621-a04c-884e002ead44\" (UID: \"d3cac855-9e22-4621-a04c-884e002ead44\") "
Oct 31 00:00:20.590006 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:20.589978    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3cac855-9e22-4621-a04c-884e002ead44-maas-api-service-ca" pod "d3cac855-9e22-4621-a04c-884e002ead44" (UID: "d3cac855-9e22-4621-a04c-884e002ead44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:20.592543 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:20.592518    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3cac855-9e22-4621-a04c-884e002ead44-kube-api-access-79fhk" pod "d3cac855-9e22-4621-a04c-884e002ead44" (UID: "d3cac855-9e22-4621-a04c-884e002ead44"). InnerVolumeSpecName "kube-api-access-79fhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:20.692636 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:20.692604    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3cac855-9e22-4621-a04c-884e002ead44-maas-api-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:20.692873 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:20.692856    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79fhk\" (UniqueName: \"kubernetes.io/projected/d3cac855-9e22-4621-a04c-884e002ead44-kube-api-access-79fhk\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:21.122991 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:21.122950    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb" podUID="4accfd35-d3c1-4b35-8271-43ffc726a5d3" containerName="payload-pre-processing" containerID="cri-o://32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc" gracePeriod=25
Oct 31 00:00:21.208448 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:21.208413    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84" path="/var/lib/kubelet/pods/0618ce43-2b19-49bb-9dd5-6813e8755b84/volumes"
Oct 31 00:00:21.208987 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:21.208968    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc" path="/var/lib/kubelet/pods/0732ac8c-6272-46d5-950b-c5d92a2216cc/volumes"
Oct 31 00:00:21.313092 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:21.313063    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"
Oct 31 00:00:21.313296 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:21.313101    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j" event={"ID":"d3cac855-9e22-4621-a04c-884e002ead44","Type":"ContainerDied","Data":"c1edad4a4051f4c3c3af72e8f354cc8800dc0deb35da471dfd4b086378c28271"}
Oct 31 00:00:21.313296 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:21.313163    4555 scope.go:122] "RemoveContainer" containerID="03f70898a549a608ad8572cdc43246517af41f0e9e0e19a033e8269e414c987f"
Oct 31 00:00:21.345977 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:21.345944    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"]
Oct 31 00:00:21.348852 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:21.348821    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ef03a-29890080-tg94j"]
Oct 31 00:00:23.208497 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:23.208462    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3cac855-9e22-4621-a04c-884e002ead44" path="/var/lib/kubelet/pods/d3cac855-9e22-4621-a04c-884e002ead44/volumes"
Oct 31 00:00:25.746681 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:25.746613    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main" containerID="cri-o://04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8" gracePeriod=45
Oct 31 00:00:26.022318 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.022288    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 31 00:00:26.172197 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.172168    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/684d4151-975d-49f1-90d5-84a476599936-tls-certs\" (UniqueName: \"kubernetes.io/secret/684d4151-975d-49f1-90d5-84a476599936-tls-certs\") pod \"684d4151-975d-49f1-90d5-84a476599936\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") "
Oct 31 00:00:26.172412 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.172239    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-model-cache\") pod \"684d4151-975d-49f1-90d5-84a476599936\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") "
Oct 31 00:00:26.172412 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.172291    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-dshm\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-dshm\") pod \"684d4151-975d-49f1-90d5-84a476599936\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") "
Oct 31 00:00:26.172412 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.172326    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-tmp-dir\") pod \"684d4151-975d-49f1-90d5-84a476599936\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") "
Oct 31 00:00:26.172628 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.172593    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-model-cache" pod "684d4151-975d-49f1-90d5-84a476599936" (UID: "684d4151-975d-49f1-90d5-84a476599936"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:26.172950 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.172915    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-tmp-dir" pod "684d4151-975d-49f1-90d5-84a476599936" (UID: "684d4151-975d-49f1-90d5-84a476599936"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:26.174578 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.174555    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-home\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-home\") pod \"684d4151-975d-49f1-90d5-84a476599936\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") "
Oct 31 00:00:26.174702 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.174618    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684d4151-975d-49f1-90d5-84a476599936-kube-api-access-d2k4m\" (UniqueName: \"kubernetes.io/projected/684d4151-975d-49f1-90d5-84a476599936-kube-api-access-d2k4m\") pod \"684d4151-975d-49f1-90d5-84a476599936\" (UID: \"684d4151-975d-49f1-90d5-84a476599936\") "
Oct 31 00:00:26.175299 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.175274    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:26.175425 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.175301    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:26.175425 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.175274    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684d4151-975d-49f1-90d5-84a476599936-tls-certs" pod "684d4151-975d-49f1-90d5-84a476599936" (UID: "684d4151-975d-49f1-90d5-84a476599936"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:00:26.175711 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.175691    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-dshm" pod "684d4151-975d-49f1-90d5-84a476599936" (UID: "684d4151-975d-49f1-90d5-84a476599936"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:26.177183 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.177159    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-home" pod "684d4151-975d-49f1-90d5-84a476599936" (UID: "684d4151-975d-49f1-90d5-84a476599936"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:00:26.177349 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.177322    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684d4151-975d-49f1-90d5-84a476599936-kube-api-access-d2k4m" pod "684d4151-975d-49f1-90d5-84a476599936" (UID: "684d4151-975d-49f1-90d5-84a476599936"). InnerVolumeSpecName "kube-api-access-d2k4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:26.277344 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.277307    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684d4151-975d-49f1-90d5-84a476599936-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:26.277344 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.277345    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:26.277575 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.277358    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684d4151-975d-49f1-90d5-84a476599936-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:26.277575 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.277390    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2k4m\" (UniqueName: \"kubernetes.io/projected/684d4151-975d-49f1-90d5-84a476599936-kube-api-access-d2k4m\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:26.356448 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.356406    4555 generic.go:356] "Generic (PLEG): container finished" podID="684d4151-975d-49f1-90d5-84a476599936" containerID="04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8" exitCode=0
Oct 31 00:00:26.356448 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.356435    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"
Oct 31 00:00:26.356679 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.356466    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" event={"ID":"684d4151-975d-49f1-90d5-84a476599936","Type":"ContainerDied","Data":"04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8"}
Oct 31 00:00:26.356679 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.356515    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl" event={"ID":"684d4151-975d-49f1-90d5-84a476599936","Type":"ContainerDied","Data":"82ce2327838ba289cb82fe0e8b7d0eda171b6e56772b624d73eb40c2d0104087"}
Oct 31 00:00:26.356679 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.356533    4555 scope.go:122] "RemoveContainer" containerID="04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8"
Oct 31 00:00:26.375977 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.375951    4555 scope.go:122] "RemoveContainer" containerID="04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8"
Oct 31 00:00:26.376273 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:26.376250    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8\": container with ID starting with 04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8 not found: ID does not exist" containerID="04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8"
Oct 31 00:00:26.376344 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.376283    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8"} err="failed to get container status \"04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8\": rpc error: code = NotFound desc = could not find container \"04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8\": container with ID starting with 04931d95b17255dea33bbb8f72ddbe676b2726bead41309dc0bdba3fdb05a7d8 not found: ID does not exist"
Oct 31 00:00:26.393140 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.393113    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"]
Oct 31 00:00:26.396209 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:26.396181    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b567bb/distinct-w3-b567bb-kserve-656b4f7bf6-lxskl"]
Oct 31 00:00:27.207101 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:27.207066    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="684d4151-975d-49f1-90d5-84a476599936" path="/var/lib/kubelet/pods/684d4151-975d-49f1-90d5-84a476599936/volumes"
Oct 31 00:00:27.448531 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:27.448487    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:00:28.150143 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:28.150105    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb" podUID="4accfd35-d3c1-4b35-8271-43ffc726a5d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:00:34.594139 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:34.594101    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"]
Oct 31 00:00:34.594697 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:34.594347    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc" podUID="6f4a512a-da26-4641-93d7-b6ae6cc73ffd" containerName="maas-api" containerID="cri-o://92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b" gracePeriod=30
Oct 31 00:00:35.045257 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.045230    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 31 00:00:35.108245 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.107517    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-metrics-certs\") pod \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") "
Oct 31 00:00:35.108245 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.107579    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-openshift-service-ca\") pod \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") "
Oct 31 00:00:35.108245 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.107646    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-kube-api-access-ghspj\" (UniqueName: \"kubernetes.io/projected/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-kube-api-access-ghspj\") pod \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") "
Oct 31 00:00:35.108245 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.107710    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-maas-api-tls\") pod \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\" (UID: \"6f4a512a-da26-4641-93d7-b6ae6cc73ffd\") "
Oct 31 00:00:35.109087 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.109057    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-openshift-service-ca" pod "6f4a512a-da26-4641-93d7-b6ae6cc73ffd" (UID: "6f4a512a-da26-4641-93d7-b6ae6cc73ffd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:35.110791 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.110757    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-metrics-certs" pod "6f4a512a-da26-4641-93d7-b6ae6cc73ffd" (UID: "6f4a512a-da26-4641-93d7-b6ae6cc73ffd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:00:35.111021 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.110992    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-maas-api-tls" pod "6f4a512a-da26-4641-93d7-b6ae6cc73ffd" (UID: "6f4a512a-da26-4641-93d7-b6ae6cc73ffd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:00:35.111628 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.111602    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-kube-api-access-ghspj" pod "6f4a512a-da26-4641-93d7-b6ae6cc73ffd" (UID: "6f4a512a-da26-4641-93d7-b6ae6cc73ffd"). InnerVolumeSpecName "kube-api-access-ghspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:35.227611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.227507    4555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-metrics-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:35.227611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.227543    4555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-openshift-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:35.227611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.227558    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghspj\" (UniqueName: \"kubernetes.io/projected/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-kube-api-access-ghspj\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:35.227611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.227573    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4a512a-da26-4641-93d7-b6ae6cc73ffd-maas-api-tls\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:35.432485 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.432446    4555 generic.go:356] "Generic (PLEG): container finished" podID="6f4a512a-da26-4641-93d7-b6ae6cc73ffd" containerID="92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b" exitCode=0
Oct 31 00:00:35.432678 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.432504    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"
Oct 31 00:00:35.432678 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.432528    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc" event={"ID":"6f4a512a-da26-4641-93d7-b6ae6cc73ffd","Type":"ContainerDied","Data":"92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b"}
Oct 31 00:00:35.432678 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.432600    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc" event={"ID":"6f4a512a-da26-4641-93d7-b6ae6cc73ffd","Type":"ContainerDied","Data":"2df77f13ac5066d449095ae89ae97ac456aea9eeccd9ca54cdfa32dde2c82118"}
Oct 31 00:00:35.432678 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.432624    4555 scope.go:122] "RemoveContainer" containerID="92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b"
Oct 31 00:00:35.444034 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.444010    4555 scope.go:122] "RemoveContainer" containerID="92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b"
Oct 31 00:00:35.444311 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:35.444288    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b\": container with ID starting with 92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b not found: ID does not exist" containerID="92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b"
Oct 31 00:00:35.444423 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.444319    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b"} err="failed to get container status \"92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b\": rpc error: code = NotFound desc = could not find container \"92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b\": container with ID starting with 92b96018897dc47f9ee6a0dee844976927eeef7d78be94a36dbfc56b4c9c6a6b not found: ID does not exist"
Oct 31 00:00:35.459482 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.459460    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"]
Oct 31 00:00:35.464323 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:35.464298    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b567bb-6df88f756d-xp5fc"]
Oct 31 00:00:37.206981 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:37.206948    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4a512a-da26-4641-93d7-b6ae6cc73ffd" path="/var/lib/kubelet/pods/6f4a512a-da26-4641-93d7-b6ae6cc73ffd/volumes"
Oct 31 00:00:38.148664 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:38.148612    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb" podUID="4accfd35-d3c1-4b35-8271-43ffc726a5d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:00:38.726296 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:38.726262    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:00:41.404198 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.404168    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 31 00:00:41.438902 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.438865    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4accfd35-d3c1-4b35-8271-43ffc726a5d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4accfd35-d3c1-4b35-8271-43ffc726a5d3-plugins-config-volume\") pod \"4accfd35-d3c1-4b35-8271-43ffc726a5d3\" (UID: \"4accfd35-d3c1-4b35-8271-43ffc726a5d3\") "
Oct 31 00:00:41.439062 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.438986    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4accfd35-d3c1-4b35-8271-43ffc726a5d3-kube-api-access-tjsjx\" (UniqueName: \"kubernetes.io/projected/4accfd35-d3c1-4b35-8271-43ffc726a5d3-kube-api-access-tjsjx\") pod \"4accfd35-d3c1-4b35-8271-43ffc726a5d3\" (UID: \"4accfd35-d3c1-4b35-8271-43ffc726a5d3\") "
Oct 31 00:00:41.440242 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.440207    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4accfd35-d3c1-4b35-8271-43ffc726a5d3-plugins-config-volume" pod "4accfd35-d3c1-4b35-8271-43ffc726a5d3" (UID: "4accfd35-d3c1-4b35-8271-43ffc726a5d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:41.442285 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.442251    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4accfd35-d3c1-4b35-8271-43ffc726a5d3-kube-api-access-tjsjx" pod "4accfd35-d3c1-4b35-8271-43ffc726a5d3" (UID: "4accfd35-d3c1-4b35-8271-43ffc726a5d3"). InnerVolumeSpecName "kube-api-access-tjsjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:41.522736 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.522700    4555 generic.go:356] "Generic (PLEG): container finished" podID="4accfd35-d3c1-4b35-8271-43ffc726a5d3" containerID="32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc" exitCode=0
Oct 31 00:00:41.522929 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.522751    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"
Oct 31 00:00:41.522929 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.522787    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb" event={"ID":"4accfd35-d3c1-4b35-8271-43ffc726a5d3","Type":"ContainerDied","Data":"32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc"}
Oct 31 00:00:41.522929 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.522841    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb" event={"ID":"4accfd35-d3c1-4b35-8271-43ffc726a5d3","Type":"ContainerDied","Data":"d4b9900b3378ba949b28eba05d19e732312a0bbe52a4bd4d21934566678423c7"}
Oct 31 00:00:41.522929 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.522865    4555 scope.go:122] "RemoveContainer" containerID="32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc"
Oct 31 00:00:41.534015 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.533997    4555 scope.go:122] "RemoveContainer" containerID="32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc"
Oct 31 00:00:41.534273 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:41.534257    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc\": container with ID starting with 32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc not found: ID does not exist" containerID="32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc"
Oct 31 00:00:41.534320 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.534281    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc"} err="failed to get container status \"32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc\": rpc error: code = NotFound desc = could not find container \"32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc\": container with ID starting with 32aaef4065a56623070b3010c5cc487229e25170dde4d8d6577f4ca35694efbc not found: ID does not exist"
Oct 31 00:00:41.540908 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.540884    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4accfd35-d3c1-4b35-8271-43ffc726a5d3-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:41.540908 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.540908    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsjx\" (UniqueName: \"kubernetes.io/projected/4accfd35-d3c1-4b35-8271-43ffc726a5d3-kube-api-access-tjsjx\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:41.556222 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.556203    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"]
Oct 31 00:00:41.561752 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:41.561731    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b567bb-76768ddbd7-mw2lb"]
Oct 31 00:00:43.218656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:43.218619    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4accfd35-d3c1-4b35-8271-43ffc726a5d3" path="/var/lib/kubelet/pods/4accfd35-d3c1-4b35-8271-43ffc726a5d3/volumes"
Oct 31 00:00:55.452361 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:55.452328    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"]
Oct 31 00:00:55.452908 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:55.452550    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy" containerID="cri-o://f5c54d3e9b8f61f80903441ec1b3cc7ec52341ff66ae88bd17c2c2d31749aad1" gracePeriod=30
Oct 31 00:00:55.460237 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:55.460199    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"]
Oct 31 00:00:55.460435 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:55.460411    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy" containerID="cri-o://8a046115cddb657a92d11791a8f8317c10853353454dd4e752a1512088ccbf94" gracePeriod=30
Oct 31 00:00:57.401520 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:57.401481    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"]
Oct 31 00:00:57.402400 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:57.402346    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy" containerID="cri-o://843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6" gracePeriod=30
Oct 31 00:00:57.410979 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:57.410953    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"]
Oct 31 00:00:57.411237 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:00:57.411188    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy" containerID="cri-o://46a46e95cbf0ce814741316fba13c7214a03bd83759646d5016d42fead150e2f" gracePeriod=30
Oct 31 00:00:57.449511 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:00:57.449470    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:01:00.679434 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.679138    4555 generic.go:356] "Generic (PLEG): container finished" podID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerID="8a046115cddb657a92d11791a8f8317c10853353454dd4e752a1512088ccbf94" exitCode=0
Oct 31 00:01:00.679434 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.679346    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" event={"ID":"99a692a4-2c57-422b-8f8e-8e8b206a0232","Type":"ContainerDied","Data":"8a046115cddb657a92d11791a8f8317c10853353454dd4e752a1512088ccbf94"}
Oct 31 00:01:00.681782 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.681597    4555 generic.go:356] "Generic (PLEG): container finished" podID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerID="f5c54d3e9b8f61f80903441ec1b3cc7ec52341ff66ae88bd17c2c2d31749aad1" exitCode=0
Oct 31 00:01:00.681782 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.681708    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" event={"ID":"caa094ff-af62-4f11-a3b9-d2f08e4d003c","Type":"ContainerDied","Data":"f5c54d3e9b8f61f80903441ec1b3cc7ec52341ff66ae88bd17c2c2d31749aad1"}
Oct 31 00:01:00.754657 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.754628    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 31 00:01:00.779123 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.779103    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 31 00:01:00.792408 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792349    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-kube-api-access-rp585\" (UniqueName: \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-kube-api-access-rp585\") pod \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") "
Oct 31 00:01:00.792569 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792445    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-certs\") pod \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") "
Oct 31 00:01:00.792569 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792526    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-socket\") pod \"99a692a4-2c57-422b-8f8e-8e8b206a0232\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") "
Oct 31 00:01:00.792705 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792572    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-credential-socket\") pod \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") "
Oct 31 00:01:00.792705 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792604    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-token\" (UniqueName: \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-token\") pod \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") "
Oct 31 00:01:00.792705 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792638    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-podinfo\") pod \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") "
Oct 31 00:01:00.792705 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792665    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-credential-socket\") pod \"99a692a4-2c57-422b-8f8e-8e8b206a0232\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") "
Oct 31 00:01:00.792931 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792715    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-envoy\") pod \"99a692a4-2c57-422b-8f8e-8e8b206a0232\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") "
Oct 31 00:01:00.792931 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792742    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-token\" (UniqueName: \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-token\") pod \"99a692a4-2c57-422b-8f8e-8e8b206a0232\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") "
Oct 31 00:01:00.792931 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792768    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istiod-ca-cert\") pod \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") "
Oct 31 00:01:00.792931 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792813    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-data\") pod \"99a692a4-2c57-422b-8f8e-8e8b206a0232\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") "
Oct 31 00:01:00.792931 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792793    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-certs" pod "caa094ff-af62-4f11-a3b9-d2f08e4d003c" (UID: "caa094ff-af62-4f11-a3b9-d2f08e4d003c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:00.792931 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792860    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-certs\") pod \"99a692a4-2c57-422b-8f8e-8e8b206a0232\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") "
Oct 31 00:01:00.792931 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792890    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-envoy\") pod \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") "
Oct 31 00:01:00.792931 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792921    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-data\") pod \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") "
Oct 31 00:01:00.793408 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792968    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-kube-api-access-pn4kh\" (UniqueName: \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-kube-api-access-pn4kh\") pod \"99a692a4-2c57-422b-8f8e-8e8b206a0232\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") "
Oct 31 00:01:00.793408 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792996    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-podinfo\") pod \"99a692a4-2c57-422b-8f8e-8e8b206a0232\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") "
Oct 31 00:01:00.793408 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.793057    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-socket\") pod \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\" (UID: \"caa094ff-af62-4f11-a3b9-d2f08e4d003c\") "
Oct 31 00:01:00.793408 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.792852    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-socket" pod "99a692a4-2c57-422b-8f8e-8e8b206a0232" (UID: "99a692a4-2c57-422b-8f8e-8e8b206a0232"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:00.793408 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.793109    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a692a4-2c57-422b-8f8e-8e8b206a0232-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a692a4-2c57-422b-8f8e-8e8b206a0232-istiod-ca-cert\") pod \"99a692a4-2c57-422b-8f8e-8e8b206a0232\" (UID: \"99a692a4-2c57-422b-8f8e-8e8b206a0232\") "
Oct 31 00:01:00.793408 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.793081    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-credential-socket" pod "99a692a4-2c57-422b-8f8e-8e8b206a0232" (UID: "99a692a4-2c57-422b-8f8e-8e8b206a0232"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:00.793801 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.793410    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-credential-socket" pod "caa094ff-af62-4f11-a3b9-d2f08e4d003c" (UID: "caa094ff-af62-4f11-a3b9-d2f08e4d003c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:00.793801 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.793781    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.793924 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.793805    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.793924 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.793823    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.793924 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.793839    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.794246 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.794222    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a692a4-2c57-422b-8f8e-8e8b206a0232-istiod-ca-cert" pod "99a692a4-2c57-422b-8f8e-8e8b206a0232" (UID: "99a692a4-2c57-422b-8f8e-8e8b206a0232"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:01:00.796056 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.796029    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-kube-api-access-rp585" pod "caa094ff-af62-4f11-a3b9-d2f08e4d003c" (UID: "caa094ff-af62-4f11-a3b9-d2f08e4d003c"). InnerVolumeSpecName "kube-api-access-rp585". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:00.796278 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.796259    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-data" pod "99a692a4-2c57-422b-8f8e-8e8b206a0232" (UID: "99a692a4-2c57-422b-8f8e-8e8b206a0232"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:00.796482 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.796462    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-podinfo" pod "99a692a4-2c57-422b-8f8e-8e8b206a0232" (UID: "99a692a4-2c57-422b-8f8e-8e8b206a0232"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:01:00.796643 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.796628    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-socket" pod "caa094ff-af62-4f11-a3b9-d2f08e4d003c" (UID: "caa094ff-af62-4f11-a3b9-d2f08e4d003c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:00.796724 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.796663    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istiod-ca-cert" pod "caa094ff-af62-4f11-a3b9-d2f08e4d003c" (UID: "caa094ff-af62-4f11-a3b9-d2f08e4d003c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:01:00.796881 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.796861    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-data" pod "caa094ff-af62-4f11-a3b9-d2f08e4d003c" (UID: "caa094ff-af62-4f11-a3b9-d2f08e4d003c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:00.797060 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.797042    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-certs" pod "99a692a4-2c57-422b-8f8e-8e8b206a0232" (UID: "99a692a4-2c57-422b-8f8e-8e8b206a0232"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:00.819038 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.818909    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-token" pod "caa094ff-af62-4f11-a3b9-d2f08e4d003c" (UID: "caa094ff-af62-4f11-a3b9-d2f08e4d003c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:00.819038 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.819007    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-kube-api-access-pn4kh" pod "99a692a4-2c57-422b-8f8e-8e8b206a0232" (UID: "99a692a4-2c57-422b-8f8e-8e8b206a0232"). InnerVolumeSpecName "kube-api-access-pn4kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:00.819336 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.819310    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-token" pod "99a692a4-2c57-422b-8f8e-8e8b206a0232" (UID: "99a692a4-2c57-422b-8f8e-8e8b206a0232"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:00.819553 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.819527    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-envoy" pod "caa094ff-af62-4f11-a3b9-d2f08e4d003c" (UID: "caa094ff-af62-4f11-a3b9-d2f08e4d003c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:00.820111 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.820063    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-envoy" pod "99a692a4-2c57-422b-8f8e-8e8b206a0232" (UID: "99a692a4-2c57-422b-8f8e-8e8b206a0232"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:00.822517 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.822490    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-podinfo" pod "caa094ff-af62-4f11-a3b9-d2f08e4d003c" (UID: "caa094ff-af62-4f11-a3b9-d2f08e4d003c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.894997    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895043    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895060    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895073    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895087    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895101    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895115    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a692a4-2c57-422b-8f8e-8e8b206a0232-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895128    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895144    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895158    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn4kh\" (UniqueName: \"kubernetes.io/projected/99a692a4-2c57-422b-8f8e-8e8b206a0232-kube-api-access-pn4kh\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895173    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a692a4-2c57-422b-8f8e-8e8b206a0232-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895187    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa094ff-af62-4f11-a3b9-d2f08e4d003c-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895200    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a692a4-2c57-422b-8f8e-8e8b206a0232-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:00.895304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:00.895214    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp585\" (UniqueName: \"kubernetes.io/projected/caa094ff-af62-4f11-a3b9-d2f08e4d003c-kube-api-access-rp585\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:01.692210 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:01.692176    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"
Oct 31 00:01:01.692210 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:01.692181    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4" event={"ID":"99a692a4-2c57-422b-8f8e-8e8b206a0232","Type":"ContainerDied","Data":"a5b39f5b9f268bdd2ac0392d308a9e24d7b1e744823a2970caf7cfb310ec74a0"}
Oct 31 00:01:01.692809 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:01.692254    4555 scope.go:122] "RemoveContainer" containerID="8a046115cddb657a92d11791a8f8317c10853353454dd4e752a1512088ccbf94"
Oct 31 00:01:01.696001 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:01.695977    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr" event={"ID":"caa094ff-af62-4f11-a3b9-d2f08e4d003c","Type":"ContainerDied","Data":"128864b7a5f6ad92820081a0f665207771c48f425a785130619f8c7ddd1971d1"}
Oct 31 00:01:01.696220 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:01.696090    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"
Oct 31 00:01:01.705357 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:01.705336    4555 scope.go:122] "RemoveContainer" containerID="f5c54d3e9b8f61f80903441ec1b3cc7ec52341ff66ae88bd17c2c2d31749aad1"
Oct 31 00:01:01.721621 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:01.721596    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"]
Oct 31 00:01:01.725401 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:01.725364    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-bwkm4"]
Oct 31 00:01:01.736584 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:01.736529    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"]
Oct 31 00:01:01.740574 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:01.740552    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843d8c-openshift-default-6d5b86bb45-5hcwr"]
Oct 31 00:01:02.701416 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.696665    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 31 00:01:02.708597 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.708562    4555 generic.go:356] "Generic (PLEG): container finished" podID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerID="843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6" exitCode=0
Oct 31 00:01:02.708757 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.708749    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"
Oct 31 00:01:02.708913 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.708894    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" event={"ID":"9dd4b677-566a-4c91-8f0f-c737e54ebf67","Type":"ContainerDied","Data":"843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6"}
Oct 31 00:01:02.709008 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.708949    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj" event={"ID":"9dd4b677-566a-4c91-8f0f-c737e54ebf67","Type":"ContainerDied","Data":"b745209323ef9a1a4ec98d0db0b9d3120ae85c257aa925b34a0df5e723c22b8b"}
Oct 31 00:01:02.709008 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.708980    4555 scope.go:122] "RemoveContainer" containerID="843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6"
Oct 31 00:01:02.711697 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.711669    4555 generic.go:356] "Generic (PLEG): container finished" podID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerID="46a46e95cbf0ce814741316fba13c7214a03bd83759646d5016d42fead150e2f" exitCode=0
Oct 31 00:01:02.711844 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.711754    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" event={"ID":"c73713bb-0f2b-4f38-8d81-bb0b437348d1","Type":"ContainerDied","Data":"46a46e95cbf0ce814741316fba13c7214a03bd83759646d5016d42fead150e2f"}
Oct 31 00:01:02.718002 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.717974    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-podinfo\") pod \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") "
Oct 31 00:01:02.718142 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718019    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-data\") pod \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") "
Oct 31 00:01:02.718142 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718041    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-certs\") pod \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") "
Oct 31 00:01:02.718142 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718067    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-token\" (UniqueName: \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-token\") pod \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") "
Oct 31 00:01:02.718142 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718100    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-credential-socket\") pod \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") "
Oct 31 00:01:02.718142 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718116    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-envoy\") pod \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") "
Oct 31 00:01:02.718348 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718162    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-kube-api-access-25sv7\" (UniqueName: \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-kube-api-access-25sv7\") pod \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") "
Oct 31 00:01:02.718348 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718207    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-socket\") pod \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") "
Oct 31 00:01:02.718348 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718253    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istiod-ca-cert\") pod \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\" (UID: \"9dd4b677-566a-4c91-8f0f-c737e54ebf67\") "
Oct 31 00:01:02.719748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718720    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-certs" pod "9dd4b677-566a-4c91-8f0f-c737e54ebf67" (UID: "9dd4b677-566a-4c91-8f0f-c737e54ebf67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:02.719748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718776    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-data" pod "9dd4b677-566a-4c91-8f0f-c737e54ebf67" (UID: "9dd4b677-566a-4c91-8f0f-c737e54ebf67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:02.719748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.718828    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-credential-socket" pod "9dd4b677-566a-4c91-8f0f-c737e54ebf67" (UID: "9dd4b677-566a-4c91-8f0f-c737e54ebf67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:02.719748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.719563    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-socket" pod "9dd4b677-566a-4c91-8f0f-c737e54ebf67" (UID: "9dd4b677-566a-4c91-8f0f-c737e54ebf67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:02.719748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.719722    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istiod-ca-cert" pod "9dd4b677-566a-4c91-8f0f-c737e54ebf67" (UID: "9dd4b677-566a-4c91-8f0f-c737e54ebf67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:01:02.721006 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.720980    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-podinfo" pod "9dd4b677-566a-4c91-8f0f-c737e54ebf67" (UID: "9dd4b677-566a-4c91-8f0f-c737e54ebf67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:01:02.721118 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.721101    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-token" pod "9dd4b677-566a-4c91-8f0f-c737e54ebf67" (UID: "9dd4b677-566a-4c91-8f0f-c737e54ebf67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:02.722152 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.722128    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-envoy" pod "9dd4b677-566a-4c91-8f0f-c737e54ebf67" (UID: "9dd4b677-566a-4c91-8f0f-c737e54ebf67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:02.724511 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.724268    4555 scope.go:122] "RemoveContainer" containerID="843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6"
Oct 31 00:01:02.724511 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.724268    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-kube-api-access-25sv7" pod "9dd4b677-566a-4c91-8f0f-c737e54ebf67" (UID: "9dd4b677-566a-4c91-8f0f-c737e54ebf67"). InnerVolumeSpecName "kube-api-access-25sv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:02.725148 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:01:02.724957    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6\": container with ID starting with 843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6 not found: ID does not exist" containerID="843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6"
Oct 31 00:01:02.725148 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.724992    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6"} err="failed to get container status \"843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6\": rpc error: code = NotFound desc = could not find container \"843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6\": container with ID starting with 843630d19fd0a18a1d5d8db0da349c7a9b84806e515b7fe0bfa503c659b8c7c6 not found: ID does not exist"
Oct 31 00:01:02.768517 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.768141    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 31 00:01:02.819998 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.819960    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-data\") pod \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") "
Oct 31 00:01:02.820199 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820056    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istiod-ca-cert\") pod \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") "
Oct 31 00:01:02.820199 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820087    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-certs\") pod \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") "
Oct 31 00:01:02.820199 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820117    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-podinfo\") pod \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") "
Oct 31 00:01:02.820458 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820430    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-socket\") pod \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") "
Oct 31 00:01:02.820554 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820518    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-kube-api-access-pljd2\" (UniqueName: \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-kube-api-access-pljd2\") pod \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") "
Oct 31 00:01:02.820554 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820534    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-data" pod "c73713bb-0f2b-4f38-8d81-bb0b437348d1" (UID: "c73713bb-0f2b-4f38-8d81-bb0b437348d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:02.820675 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820606    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-credential-socket\") pod \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") "
Oct 31 00:01:02.820675 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820635    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-token\" (UniqueName: \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-token\") pod \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") "
Oct 31 00:01:02.820778 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820681    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-envoy\") pod \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\" (UID: \"c73713bb-0f2b-4f38-8d81-bb0b437348d1\") "
Oct 31 00:01:02.820778 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820754    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-certs" pod "c73713bb-0f2b-4f38-8d81-bb0b437348d1" (UID: "c73713bb-0f2b-4f38-8d81-bb0b437348d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:02.820778 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.820755    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istiod-ca-cert" pod "c73713bb-0f2b-4f38-8d81-bb0b437348d1" (UID: "c73713bb-0f2b-4f38-8d81-bb0b437348d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:01:02.821070 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821051    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-socket" pod "c73713bb-0f2b-4f38-8d81-bb0b437348d1" (UID: "c73713bb-0f2b-4f38-8d81-bb0b437348d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:02.821245 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821224    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-credential-socket" pod "c73713bb-0f2b-4f38-8d81-bb0b437348d1" (UID: "c73713bb-0f2b-4f38-8d81-bb0b437348d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:02.821410 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821385    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821413    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25sv7\" (UniqueName: \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-kube-api-access-25sv7\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821430    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821443    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821456    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821468    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821481    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821494    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821508    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821522    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821535    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821548    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821561    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.821659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.821575    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dd4b677-566a-4c91-8f0f-c737e54ebf67-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.823255 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.823229    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-podinfo" pod "c73713bb-0f2b-4f38-8d81-bb0b437348d1" (UID: "c73713bb-0f2b-4f38-8d81-bb0b437348d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:01:02.823489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.823471    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-token" pod "c73713bb-0f2b-4f38-8d81-bb0b437348d1" (UID: "c73713bb-0f2b-4f38-8d81-bb0b437348d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:02.823655 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.823639    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-kube-api-access-pljd2" pod "c73713bb-0f2b-4f38-8d81-bb0b437348d1" (UID: "c73713bb-0f2b-4f38-8d81-bb0b437348d1"). InnerVolumeSpecName "kube-api-access-pljd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:02.823777 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.823762    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-envoy" pod "c73713bb-0f2b-4f38-8d81-bb0b437348d1" (UID: "c73713bb-0f2b-4f38-8d81-bb0b437348d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:02.923171 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.923070    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.923171 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.923105    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pljd2\" (UniqueName: \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-kube-api-access-pljd2\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.923171 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.923119    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:02.923171 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:02.923132    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73713bb-0f2b-4f38-8d81-bb0b437348d1-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:03.034597 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.034564    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"]
Oct 31 00:01:03.041405 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.041345    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-tgxdj"]
Oct 31 00:01:03.209650 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.209553    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" path="/var/lib/kubelet/pods/99a692a4-2c57-422b-8f8e-8e8b206a0232/volumes"
Oct 31 00:01:03.210174 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.210151    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" path="/var/lib/kubelet/pods/9dd4b677-566a-4c91-8f0f-c737e54ebf67/volumes"
Oct 31 00:01:03.210744 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.210719    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" path="/var/lib/kubelet/pods/caa094ff-af62-4f11-a3b9-d2f08e4d003c/volumes"
Oct 31 00:01:03.726329 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.726295    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"
Oct 31 00:01:03.726329 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.726309    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t" event={"ID":"c73713bb-0f2b-4f38-8d81-bb0b437348d1","Type":"ContainerDied","Data":"b45b31d30a69561073e89f5e3a9946bcc94d9d67b1d6d529ccd3ead242007f2b"}
Oct 31 00:01:03.726921 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.726405    4555 scope.go:122] "RemoveContainer" containerID="46a46e95cbf0ce814741316fba13c7214a03bd83759646d5016d42fead150e2f"
Oct 31 00:01:03.750156 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.750119    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"]
Oct 31 00:01:03.753781 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.753758    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b567bb-openshift-default-f5f7c68ff-7hf9t"]
Oct 31 00:01:03.903982 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:03.903946    4555 scope.go:122] "RemoveContainer" containerID="8928a1a4fb3b6ace4c20f81500bc72868777ececbcdad0cda5e33968c02ae294"
Oct 31 00:01:05.207826 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:05.207785    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" path="/var/lib/kubelet/pods/c73713bb-0f2b-4f38-8d81-bb0b437348d1/volumes"
Oct 31 00:01:05.294141 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:05.294100    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:01:17.949642 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:17.949586    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"]
Oct 31 00:01:17.950141 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:17.949955    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy" containerID="cri-o://9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a" gracePeriod=30
Oct 31 00:01:23.232307 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.232277    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 31 00:01:23.298277 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298240    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istiod-ca-cert\") pod \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") "
Oct 31 00:01:23.298510 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298316    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-credential-socket\") pod \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") "
Oct 31 00:01:23.298510 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298397    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-envoy\") pod \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") "
Oct 31 00:01:23.298510 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298442    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-certs\") pod \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") "
Oct 31 00:01:23.298510 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298493    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-podinfo\") pod \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") "
Oct 31 00:01:23.298757 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298529    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-token\" (UniqueName: \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-token\") pod \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") "
Oct 31 00:01:23.298757 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298657    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-kube-api-access-hz8qc\" (UniqueName: \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-kube-api-access-hz8qc\") pod \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") "
Oct 31 00:01:23.298757 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298688    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istiod-ca-cert" pod "76874aeb-95f4-483e-acca-cb1ba0b9ecfc" (UID: "76874aeb-95f4-483e-acca-cb1ba0b9ecfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:01:23.298757 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298698    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-data\") pod \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") "
Oct 31 00:01:23.298985 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298785    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-socket\") pod \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\" (UID: \"76874aeb-95f4-483e-acca-cb1ba0b9ecfc\") "
Oct 31 00:01:23.298985 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.298960    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-data" pod "76874aeb-95f4-483e-acca-cb1ba0b9ecfc" (UID: "76874aeb-95f4-483e-acca-cb1ba0b9ecfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:23.299520 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.299497    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:23.299691 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.299674    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:23.300023 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.299992    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-certs" pod "76874aeb-95f4-483e-acca-cb1ba0b9ecfc" (UID: "76874aeb-95f4-483e-acca-cb1ba0b9ecfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:23.302498 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.302468    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-socket" pod "76874aeb-95f4-483e-acca-cb1ba0b9ecfc" (UID: "76874aeb-95f4-483e-acca-cb1ba0b9ecfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:23.302728 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.302703    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-credential-socket" pod "76874aeb-95f4-483e-acca-cb1ba0b9ecfc" (UID: "76874aeb-95f4-483e-acca-cb1ba0b9ecfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:23.304856 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.303027    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-envoy" pod "76874aeb-95f4-483e-acca-cb1ba0b9ecfc" (UID: "76874aeb-95f4-483e-acca-cb1ba0b9ecfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:23.304856 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.304788    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-token" pod "76874aeb-95f4-483e-acca-cb1ba0b9ecfc" (UID: "76874aeb-95f4-483e-acca-cb1ba0b9ecfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:23.305485 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.305441    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-kube-api-access-hz8qc" pod "76874aeb-95f4-483e-acca-cb1ba0b9ecfc" (UID: "76874aeb-95f4-483e-acca-cb1ba0b9ecfc"). InnerVolumeSpecName "kube-api-access-hz8qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:23.306742 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.306717    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-podinfo" pod "76874aeb-95f4-483e-acca-cb1ba0b9ecfc" (UID: "76874aeb-95f4-483e-acca-cb1ba0b9ecfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:01:23.400466 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.400435    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:23.400466 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.400465    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:23.400731 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.400479    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:23.400731 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.400488    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:23.400731 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.400502    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz8qc\" (UniqueName: \"kubernetes.io/projected/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-kube-api-access-hz8qc\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:23.400731 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.400516    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:23.400731 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.400529    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76874aeb-95f4-483e-acca-cb1ba0b9ecfc-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:23.885416 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.885346    4555 generic.go:356] "Generic (PLEG): container finished" podID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerID="9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a" exitCode=0
Oct 31 00:01:23.885618 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.885439    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" event={"ID":"76874aeb-95f4-483e-acca-cb1ba0b9ecfc","Type":"ContainerDied","Data":"9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a"}
Oct 31 00:01:23.885618 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.885517    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8" event={"ID":"76874aeb-95f4-483e-acca-cb1ba0b9ecfc","Type":"ContainerDied","Data":"c9e257ed7f1b8e5aa5cfceb98f3145cab9bd1fa0a4ff3281a3ba2f7c6331b668"}
Oct 31 00:01:23.885618 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.885548    4555 scope.go:122] "RemoveContainer" containerID="9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a"
Oct 31 00:01:23.885816 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.885450    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"
Oct 31 00:01:23.897379 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.897348    4555 scope.go:122] "RemoveContainer" containerID="03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01"
Oct 31 00:01:23.912224 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.912195    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"]
Oct 31 00:01:23.912427 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.912338    4555 scope.go:122] "RemoveContainer" containerID="9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a"
Oct 31 00:01:23.912775 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:01:23.912752    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a\": container with ID starting with 9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a not found: ID does not exist" containerID="9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a"
Oct 31 00:01:23.912880 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.912780    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a"} err="failed to get container status \"9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a\": rpc error: code = NotFound desc = could not find container \"9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a\": container with ID starting with 9aedf44a7e1e98d6d2eca9ec02447764c7761b15060451afd13ce4345f26fc6a not found: ID does not exist"
Oct 31 00:01:23.912880 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.912801    4555 scope.go:122] "RemoveContainer" containerID="03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01"
Oct 31 00:01:23.913095 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:01:23.913076    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01\": container with ID starting with 03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01 not found: ID does not exist" containerID="03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01"
Oct 31 00:01:23.913160 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.913104    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01"} err="failed to get container status \"03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01\": rpc error: code = NotFound desc = could not find container \"03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01\": container with ID starting with 03194942872b96a676fde5228347b381a16126ff1222634118a8b4499cfc5e01 not found: ID does not exist"
Oct 31 00:01:23.919441 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:23.919414    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-885999-openshift-default-6888c4cb57-knds8"]
Oct 31 00:01:25.208295 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:25.208255    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" path="/var/lib/kubelet/pods/76874aeb-95f4-483e-acca-cb1ba0b9ecfc/volumes"
Oct 31 00:01:27.350707 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:27.350672    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"]
Oct 31 00:01:27.363479 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:27.363450    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"]
Oct 31 00:01:27.447053 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:01:27.447014    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:01:30.552505 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:30.552470    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:01:32.692320 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:32.692276    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"]
Oct 31 00:01:33.177482 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.177444    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"]
Oct 31 00:01:33.178136 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178115    4555 cpu_manager.go:414] "containerMap: removing container" podUID="727002c6-942f-4ea0-b31e-74acc34261d6" containerName="cleanup"
Oct 31 00:01:33.178186 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178141    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727002c6-942f-4ea0-b31e-74acc34261d6" containerName="cleanup"
Oct 31 00:01:33.178186 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178158    4555 cpu_manager.go:414] "containerMap: removing container" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy"
Oct 31 00:01:33.178186 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178166    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178184    4555 cpu_manager.go:414] "containerMap: removing container" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178193    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178203    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerName="cleanup"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178210    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerName="cleanup"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178226    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerName="cleanup"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178235    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerName="cleanup"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178252    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerName="cleanup"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178260    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerName="cleanup"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178270    4555 cpu_manager.go:414] "containerMap: removing container" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178277    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178286    4555 cpu_manager.go:414] "containerMap: removing container" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178295    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178312    4555 cpu_manager.go:414] "containerMap: removing container" podUID="6f4a512a-da26-4641-93d7-b6ae6cc73ffd" containerName="maas-api"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178319    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4a512a-da26-4641-93d7-b6ae6cc73ffd" containerName="maas-api"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178334    4555 cpu_manager.go:414] "containerMap: removing container" podUID="d3cac855-9e22-4621-a04c-884e002ead44" containerName="cleanup"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178341    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3cac855-9e22-4621-a04c-884e002ead44" containerName="cleanup"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178351    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerName="cleanup"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178410    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerName="cleanup"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178447    4555 cpu_manager.go:414] "containerMap: removing container" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178455    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178469    4555 cpu_manager.go:414] "containerMap: removing container" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178476    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178492    4555 cpu_manager.go:414] "containerMap: removing container" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy"
Oct 31 00:01:33.178489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178499    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178510    4555 cpu_manager.go:414] "containerMap: removing container" podUID="2918e4a9-cdfb-460f-82f5-941944ce7a47" containerName="payload-processing"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178517    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2918e4a9-cdfb-460f-82f5-941944ce7a47" containerName="payload-processing"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178527    4555 cpu_manager.go:414] "containerMap: removing container" podUID="d3cac855-9e22-4621-a04c-884e002ead44" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178534    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3cac855-9e22-4621-a04c-884e002ead44" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178543    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178552    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178562    4555 cpu_manager.go:414] "containerMap: removing container" podUID="d3cac855-9e22-4621-a04c-884e002ead44" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178569    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3cac855-9e22-4621-a04c-884e002ead44" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178597    4555 cpu_manager.go:414] "containerMap: removing container" podUID="4accfd35-d3c1-4b35-8271-43ffc726a5d3" containerName="payload-pre-processing"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178604    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4accfd35-d3c1-4b35-8271-43ffc726a5d3" containerName="payload-pre-processing"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178617    4555 cpu_manager.go:414] "containerMap: removing container" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178624    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178632    4555 cpu_manager.go:414] "containerMap: removing container" podUID="a1c131d4-4156-40bf-9f1e-e62079bd51a8" containerName="maas-api"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178641    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c131d4-4156-40bf-9f1e-e62079bd51a8" containerName="maas-api"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178657    4555 cpu_manager.go:414] "containerMap: removing container" podUID="727002c6-942f-4ea0-b31e-74acc34261d6" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178664    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727002c6-942f-4ea0-b31e-74acc34261d6" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178677    4555 cpu_manager.go:414] "containerMap: removing container" podUID="b3cd0b30-12b8-4c3a-b1ca-dba8525e7190" containerName="payload-pre-processing"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178685    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cd0b30-12b8-4c3a-b1ca-dba8525e7190" containerName="payload-pre-processing"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178697    4555 cpu_manager.go:414] "containerMap: removing container" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178704    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178833    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178845    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="727002c6-942f-4ea0-b31e-74acc34261d6" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178857    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cd0b30-12b8-4c3a-b1ca-dba8525e7190" containerName="payload-pre-processing"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178868    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178877    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa094ff-af62-4f11-a3b9-d2f08e4d003c" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178885    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="727002c6-942f-4ea0-b31e-74acc34261d6" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178894    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd4b677-566a-4c91-8f0f-c737e54ebf67" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178904    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73713bb-0f2b-4f38-8d81-bb0b437348d1" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178914    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178924    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178935    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="727002c6-942f-4ea0-b31e-74acc34261d6" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178946    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178954    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c6ddb0-5235-4458-a035-b94f0588a3bf" containerName="main"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178962    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178973    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4a512a-da26-4641-93d7-b6ae6cc73ffd" containerName="maas-api"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178983    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3cac855-9e22-4621-a04c-884e002ead44" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.178991    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179000    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a692a4-2c57-422b-8f8e-8e8b206a0232" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179008    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="c907bcbf-31c9-472f-97c2-58b87c67e7f1" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179016    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179025    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="2918e4a9-cdfb-460f-82f5-941944ce7a47" containerName="payload-processing"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179034    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3cac855-9e22-4621-a04c-884e002ead44" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179043    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0732ac8c-6272-46d5-950b-c5d92a2216cc" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179051    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3cac855-9e22-4621-a04c-884e002ead44" containerName="cleanup"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179059    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179070    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="4accfd35-d3c1-4b35-8271-43ffc726a5d3" containerName="payload-pre-processing"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179079    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="684d4151-975d-49f1-90d5-84a476599936" containerName="main"
Oct 31 00:01:33.179402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.179087    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c131d4-4156-40bf-9f1e-e62079bd51a8" containerName="maas-api"
Oct 31 00:01:33.182971 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.182951    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.192146 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.192116    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"]
Oct 31 00:01:33.206751 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.206703    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ef74bfe-pjwmz\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.206928 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.206821    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5tjk\" (UniqueName: \"kubernetes.io/projected/1493b5fd-8940-438b-9063-81aa1ae5a1c3-kube-api-access-f5tjk\") pod \"maas-api-revoke-sub-0ef74bfe-pjwmz\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.206998 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.206966    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ef74bfe-pjwmz\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.241590 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.241551    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:01:33.241590 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.241590    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 31 00:01:33.308233 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.308194    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ef74bfe-pjwmz\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.308461 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.308257    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5tjk\" (UniqueName: \"kubernetes.io/projected/1493b5fd-8940-438b-9063-81aa1ae5a1c3-kube-api-access-f5tjk\") pod \"maas-api-revoke-sub-0ef74bfe-pjwmz\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.308461 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.308396    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ef74bfe-pjwmz\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.308972 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.308933    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ef74bfe-pjwmz\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.310889 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.310868    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ef74bfe-pjwmz\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.321489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.321462    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5tjk\" (UniqueName: \"kubernetes.io/projected/1493b5fd-8940-438b-9063-81aa1ae5a1c3-kube-api-access-f5tjk\") pod \"maas-api-revoke-sub-0ef74bfe-pjwmz\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.501236 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.501144    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:33.788347 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.788305    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"]
Oct 31 00:01:33.798325 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:01:33.798297    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1493b5fd_8940_438b_9063_81aa1ae5a1c3.slice/crio-7d8256722091458aeae467fcf515ec14228cb83c867f9e906c9ed288059b9abf WatchSource:0}: Error finding container 7d8256722091458aeae467fcf515ec14228cb83c867f9e906c9ed288059b9abf: Status 404 returned error can't find the container with id 7d8256722091458aeae467fcf515ec14228cb83c867f9e906c9ed288059b9abf
Oct 31 00:01:33.957811 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:33.957778    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz" event={"ID":"1493b5fd-8940-438b-9063-81aa1ae5a1c3","Type":"ContainerStarted","Data":"7d8256722091458aeae467fcf515ec14228cb83c867f9e906c9ed288059b9abf"}
Oct 31 00:01:34.967144 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:34.967105    4555 generic.go:356] "Generic (PLEG): container finished" podID="1493b5fd-8940-438b-9063-81aa1ae5a1c3" containerID="d3a19f2b7975fbc3ec15c9ef675d34cb800470ce1ba0e85cb913f20f6ba042d1" exitCode=0
Oct 31 00:01:34.967611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:34.967232    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz" event={"ID":"1493b5fd-8940-438b-9063-81aa1ae5a1c3","Type":"ContainerDied","Data":"d3a19f2b7975fbc3ec15c9ef675d34cb800470ce1ba0e85cb913f20f6ba042d1"}
Oct 31 00:01:36.165502 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.165471    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:36.245289 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.245194    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1493b5fd-8940-438b-9063-81aa1ae5a1c3-kube-api-access-f5tjk\" (UniqueName: \"kubernetes.io/projected/1493b5fd-8940-438b-9063-81aa1ae5a1c3-kube-api-access-f5tjk\") pod \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") "
Oct 31 00:01:36.245289 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.245289    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-openshift-ca\") pod \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") "
Oct 31 00:01:36.245593 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.245359    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-xks-ca\") pod \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\" (UID: \"1493b5fd-8940-438b-9063-81aa1ae5a1c3\") "
Oct 31 00:01:36.245733 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.245708    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-openshift-ca" pod "1493b5fd-8940-438b-9063-81aa1ae5a1c3" (UID: "1493b5fd-8940-438b-9063-81aa1ae5a1c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:01:36.245853 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.245837    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:36.247521 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.247499    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-xks-ca" pod "1493b5fd-8940-438b-9063-81aa1ae5a1c3" (UID: "1493b5fd-8940-438b-9063-81aa1ae5a1c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:01:36.247623 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.247534    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1493b5fd-8940-438b-9063-81aa1ae5a1c3-kube-api-access-f5tjk" pod "1493b5fd-8940-438b-9063-81aa1ae5a1c3" (UID: "1493b5fd-8940-438b-9063-81aa1ae5a1c3"). InnerVolumeSpecName "kube-api-access-f5tjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:36.346832 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.346792    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5tjk\" (UniqueName: \"kubernetes.io/projected/1493b5fd-8940-438b-9063-81aa1ae5a1c3-kube-api-access-f5tjk\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:36.346832 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.346823    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1493b5fd-8940-438b-9063-81aa1ae5a1c3-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:36.983865 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.983827    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"
Oct 31 00:01:36.983865 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.983850    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz" event={"ID":"1493b5fd-8940-438b-9063-81aa1ae5a1c3","Type":"ContainerDied","Data":"7d8256722091458aeae467fcf515ec14228cb83c867f9e906c9ed288059b9abf"}
Oct 31 00:01:36.984111 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:36.983917    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d8256722091458aeae467fcf515ec14228cb83c867f9e906c9ed288059b9abf"
Oct 31 00:01:37.693301 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:37.693253    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk" podUID="5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6" containerName="payload-processing" containerID="cri-o://04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6" gracePeriod=25
Oct 31 00:01:42.372561 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.372449    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main" containerID="cri-o://9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344" gracePeriod=45
Oct 31 00:01:42.386889 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.386849    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" containerID="cri-o://1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1" gracePeriod=45
Oct 31 00:01:42.666695 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.666663    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 31 00:01:42.670280 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.670246    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 31 00:01:42.724351 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724316    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tmp-dir\") pod \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") "
Oct 31 00:01:42.724552 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724396    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-model-cache\") pod \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") "
Oct 31 00:01:42.724552 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724455    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-model-cache\") pod \"ddac194a-9d84-41d6-b40b-e7aa74945352\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") "
Oct 31 00:01:42.724552 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724508    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-dshm\") pod \"ddac194a-9d84-41d6-b40b-e7aa74945352\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") "
Oct 31 00:01:42.724738 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724553    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-home\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-home\") pod \"ddac194a-9d84-41d6-b40b-e7aa74945352\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") "
Oct 31 00:01:42.724738 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724585    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddac194a-9d84-41d6-b40b-e7aa74945352-tls-certs\" (UniqueName: \"kubernetes.io/secret/ddac194a-9d84-41d6-b40b-e7aa74945352-tls-certs\") pod \"ddac194a-9d84-41d6-b40b-e7aa74945352\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") "
Oct 31 00:01:42.724738 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724645    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tmp-dir" pod "16bfafce-6391-4dc5-8eb4-2a77f0f22164" (UID: "16bfafce-6391-4dc5-8eb4-2a77f0f22164"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:42.724738 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724681    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddac194a-9d84-41d6-b40b-e7aa74945352-kube-api-access-zccrc\" (UniqueName: \"kubernetes.io/projected/ddac194a-9d84-41d6-b40b-e7aa74945352-kube-api-access-zccrc\") pod \"ddac194a-9d84-41d6-b40b-e7aa74945352\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") "
Oct 31 00:01:42.724738 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724723    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16bfafce-6391-4dc5-8eb4-2a77f0f22164-kube-api-access-c5w5v\" (UniqueName: \"kubernetes.io/projected/16bfafce-6391-4dc5-8eb4-2a77f0f22164-kube-api-access-c5w5v\") pod \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") "
Oct 31 00:01:42.725035 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724792    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-dshm\") pod \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") "
Oct 31 00:01:42.725035 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724823    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tls-certs\" (UniqueName: \"kubernetes.io/secret/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tls-certs\") pod \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") "
Oct 31 00:01:42.725035 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724881    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-home\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-home\") pod \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\" (UID: \"16bfafce-6391-4dc5-8eb4-2a77f0f22164\") "
Oct 31 00:01:42.725035 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724915    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-tmp-dir\") pod \"ddac194a-9d84-41d6-b40b-e7aa74945352\" (UID: \"ddac194a-9d84-41d6-b40b-e7aa74945352\") "
Oct 31 00:01:42.725868 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.724719    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-model-cache" pod "16bfafce-6391-4dc5-8eb4-2a77f0f22164" (UID: "16bfafce-6391-4dc5-8eb4-2a77f0f22164"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:42.726662 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.726622    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-home" pod "16bfafce-6391-4dc5-8eb4-2a77f0f22164" (UID: "16bfafce-6391-4dc5-8eb4-2a77f0f22164"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:42.726869 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.726740    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-tmp-dir" pod "ddac194a-9d84-41d6-b40b-e7aa74945352" (UID: "ddac194a-9d84-41d6-b40b-e7aa74945352"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:42.734875 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.733534    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddac194a-9d84-41d6-b40b-e7aa74945352-tls-certs" pod "ddac194a-9d84-41d6-b40b-e7aa74945352" (UID: "ddac194a-9d84-41d6-b40b-e7aa74945352"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:01:42.734875 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.734767    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.734875 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.734839    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.734875 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.734836    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-model-cache" pod "ddac194a-9d84-41d6-b40b-e7aa74945352" (UID: "ddac194a-9d84-41d6-b40b-e7aa74945352"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:42.735187 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.734678    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-home" pod "ddac194a-9d84-41d6-b40b-e7aa74945352" (UID: "ddac194a-9d84-41d6-b40b-e7aa74945352"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:42.735187 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.734853    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.735187 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.734955    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.735187 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.734972    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddac194a-9d84-41d6-b40b-e7aa74945352-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.737538 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.737509    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-dshm" pod "16bfafce-6391-4dc5-8eb4-2a77f0f22164" (UID: "16bfafce-6391-4dc5-8eb4-2a77f0f22164"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:42.737648 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.737595    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddac194a-9d84-41d6-b40b-e7aa74945352-kube-api-access-zccrc" pod "ddac194a-9d84-41d6-b40b-e7aa74945352" (UID: "ddac194a-9d84-41d6-b40b-e7aa74945352"). InnerVolumeSpecName "kube-api-access-zccrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:42.738566 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.738543    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tls-certs" pod "16bfafce-6391-4dc5-8eb4-2a77f0f22164" (UID: "16bfafce-6391-4dc5-8eb4-2a77f0f22164"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:01:42.739522 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.739496    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-dshm" pod "ddac194a-9d84-41d6-b40b-e7aa74945352" (UID: "ddac194a-9d84-41d6-b40b-e7aa74945352"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:01:42.741685 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.741656    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bfafce-6391-4dc5-8eb4-2a77f0f22164-kube-api-access-c5w5v" pod "16bfafce-6391-4dc5-8eb4-2a77f0f22164" (UID: "16bfafce-6391-4dc5-8eb4-2a77f0f22164"). InnerVolumeSpecName "kube-api-access-c5w5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:42.835603 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.835573    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5w5v\" (UniqueName: \"kubernetes.io/projected/16bfafce-6391-4dc5-8eb4-2a77f0f22164-kube-api-access-c5w5v\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.835603 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.835599    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16bfafce-6391-4dc5-8eb4-2a77f0f22164-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.835603 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.835610    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bfafce-6391-4dc5-8eb4-2a77f0f22164-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.835869 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.835622    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.835869 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.835632    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.835869 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.835640    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddac194a-9d84-41d6-b40b-e7aa74945352-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:42.835869 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:42.835647    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zccrc\" (UniqueName: \"kubernetes.io/projected/ddac194a-9d84-41d6-b40b-e7aa74945352-kube-api-access-zccrc\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:43.029839 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.029806    4555 generic.go:356] "Generic (PLEG): container finished" podID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerID="9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344" exitCode=0
Oct 31 00:01:43.030017 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.029853    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"
Oct 31 00:01:43.030017 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.029894    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" event={"ID":"16bfafce-6391-4dc5-8eb4-2a77f0f22164","Type":"ContainerDied","Data":"9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344"}
Oct 31 00:01:43.030017 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.029934    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm" event={"ID":"16bfafce-6391-4dc5-8eb4-2a77f0f22164","Type":"ContainerDied","Data":"71acea1e4045fb8c0b88dad0807545b59f34c4be51dc8e368586dae350dbe7bd"}
Oct 31 00:01:43.030017 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.029957    4555 scope.go:122] "RemoveContainer" containerID="9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344"
Oct 31 00:01:43.031759 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.031736    4555 generic.go:356] "Generic (PLEG): container finished" podID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerID="1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1" exitCode=0
Oct 31 00:01:43.031901 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.031846    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" event={"ID":"ddac194a-9d84-41d6-b40b-e7aa74945352","Type":"ContainerDied","Data":"1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1"}
Oct 31 00:01:43.031901 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.031876    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" event={"ID":"ddac194a-9d84-41d6-b40b-e7aa74945352","Type":"ContainerDied","Data":"3a2ad81811396fa6f9a758e46d252a0dbb0dddd754f4e3af714b371dac3f346a"}
Oct 31 00:01:43.032048 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.031945    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"
Oct 31 00:01:43.041990 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.041833    4555 scope.go:122] "RemoveContainer" containerID="9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344"
Oct 31 00:01:43.042248 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:01:43.042226    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344\": container with ID starting with 9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344 not found: ID does not exist" containerID="9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344"
Oct 31 00:01:43.042343 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.042259    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344"} err="failed to get container status \"9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344\": rpc error: code = NotFound desc = could not find container \"9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344\": container with ID starting with 9e9a35819e49dcc3e7b7f8977354fdda11a7a1cc6632237960d025301e389344 not found: ID does not exist"
Oct 31 00:01:43.042343 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.042282    4555 scope.go:122] "RemoveContainer" containerID="1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1"
Oct 31 00:01:43.052168 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.052154    4555 scope.go:122] "RemoveContainer" containerID="1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1"
Oct 31 00:01:43.052421 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:01:43.052402    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1\": container with ID starting with 1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1 not found: ID does not exist" containerID="1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1"
Oct 31 00:01:43.052494 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.052427    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1"} err="failed to get container status \"1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1\": rpc error: code = NotFound desc = could not find container \"1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1\": container with ID starting with 1a4415948c59b8b1ffb61afd995fb074171b58619a2822408e7295f9dbead6e1 not found: ID does not exist"
Oct 31 00:01:43.062837 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.062813    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"]
Oct 31 00:01:43.066952 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.066934    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8"]
Oct 31 00:01:43.077438 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.077419    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"]
Oct 31 00:01:43.085990 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.085973    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9ef03a/distinct-w2-9ef03a-kserve-6b898876c-m69sm"]
Oct 31 00:01:43.206256 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.206223    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" path="/var/lib/kubelet/pods/16bfafce-6391-4dc5-8eb4-2a77f0f22164/volumes"
Oct 31 00:01:43.206714 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.206698    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" path="/var/lib/kubelet/pods/ddac194a-9d84-41d6-b40b-e7aa74945352/volumes"
Oct 31 00:01:43.589062 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:43.589020    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ef03a/premium-w2-9ef03a-kserve-564b59bf56-8jdr8" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:01:47.141850 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:47.141811    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk" podUID="5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:01:52.858002 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:52.857958    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:01:57.112068 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:57.112032    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"]
Oct 31 00:01:57.112926 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:57.112868    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy" containerID="cri-o://7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9" gracePeriod=30
Oct 31 00:01:57.141968 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:57.141931    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk" podUID="5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:01:57.454875 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:01:57.451404    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:01:57.963788 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:57.963731    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 31 00:01:58.043842 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.043809    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-plugins-config-volume\") pod \"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6\" (UID: \"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6\") "
Oct 31 00:01:58.044025 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.043914    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-kube-api-access-skbmj\" (UniqueName: \"kubernetes.io/projected/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-kube-api-access-skbmj\") pod \"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6\" (UID: \"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6\") "
Oct 31 00:01:58.044208 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.044186    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-plugins-config-volume" pod "5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6" (UID: "5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:01:58.046225 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.046205    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-kube-api-access-skbmj" pod "5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6" (UID: "5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6"). InnerVolumeSpecName "kube-api-access-skbmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:58.145273 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.145230    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:58.145273 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.145270    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skbmj\" (UniqueName: \"kubernetes.io/projected/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6-kube-api-access-skbmj\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:01:58.194533 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.194501    4555 generic.go:356] "Generic (PLEG): container finished" podID="5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6" containerID="04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6" exitCode=0
Oct 31 00:01:58.194718 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.194563    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk" event={"ID":"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6","Type":"ContainerDied","Data":"04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6"}
Oct 31 00:01:58.194718 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.194567    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"
Oct 31 00:01:58.194718 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.194606    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk" event={"ID":"5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6","Type":"ContainerDied","Data":"4dd48eeccc34be522d7d197df49b79b993b36ac4e6bd43342a470c888491de10"}
Oct 31 00:01:58.194718 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.194628    4555 scope.go:122] "RemoveContainer" containerID="04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6"
Oct 31 00:01:58.205531 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.205417    4555 scope.go:122] "RemoveContainer" containerID="04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6"
Oct 31 00:01:58.205911 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:01:58.205881    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6\": container with ID starting with 04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6 not found: ID does not exist" containerID="04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6"
Oct 31 00:01:58.206022 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.205922    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6"} err="failed to get container status \"04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6\": rpc error: code = NotFound desc = could not find container \"04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6\": container with ID starting with 04d8796b1367f9c96e2bb340c0ebfe203eebaf6e289f352fea919cc52fd42dd6 not found: ID does not exist"
Oct 31 00:01:58.220898 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.220828    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"]
Oct 31 00:01:58.226044 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.226021    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ef03a-6b5b6dd9b7-775hk"]
Oct 31 00:01:58.397391 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.396822    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"]
Oct 31 00:01:58.397391 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.397076    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy" containerID="cri-o://7de9401a7a9db0f0a196272a9934fbcfff026a9634e8efe1fb7da4ebc282651a" gracePeriod=30
Oct 31 00:01:58.404846 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.404813    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"]
Oct 31 00:01:58.405550 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:58.405511    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy" containerID="cri-o://aafd2075a3546fc21990cd7846023d48f4782c9384a3f11262689bc69cbf90ef" gracePeriod=30
Oct 31 00:01:59.211622 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:59.211587    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6" path="/var/lib/kubelet/pods/5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6/volumes"
Oct 31 00:01:59.348988 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:01:59.348954    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:02:02.384979 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.384905    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 31 00:02:02.484235 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.484199    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-kube-api-access-wq9zl\" (UniqueName: \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-kube-api-access-wq9zl\") pod \"cde9335c-c895-4cd1-a70b-5823203f84b0\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") "
Oct 31 00:02:02.484461 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.484261    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-certs\") pod \"cde9335c-c895-4cd1-a70b-5823203f84b0\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") "
Oct 31 00:02:02.484461 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.484329    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde9335c-c895-4cd1-a70b-5823203f84b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde9335c-c895-4cd1-a70b-5823203f84b0-istiod-ca-cert\") pod \"cde9335c-c895-4cd1-a70b-5823203f84b0\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") "
Oct 31 00:02:02.484461 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.484408    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-data\") pod \"cde9335c-c895-4cd1-a70b-5823203f84b0\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") "
Oct 31 00:02:02.484461 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.484459    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-socket\") pod \"cde9335c-c895-4cd1-a70b-5823203f84b0\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") "
Oct 31 00:02:02.484725 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.484529    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-envoy\") pod \"cde9335c-c895-4cd1-a70b-5823203f84b0\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") "
Oct 31 00:02:02.484725 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.484572    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-podinfo\") pod \"cde9335c-c895-4cd1-a70b-5823203f84b0\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") "
Oct 31 00:02:02.484725 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.484610    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-token\" (UniqueName: \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-token\") pod \"cde9335c-c895-4cd1-a70b-5823203f84b0\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") "
Oct 31 00:02:02.484725 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.484672    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-credential-socket\") pod \"cde9335c-c895-4cd1-a70b-5823203f84b0\" (UID: \"cde9335c-c895-4cd1-a70b-5823203f84b0\") "
Oct 31 00:02:02.485942 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.485286    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-socket" pod "cde9335c-c895-4cd1-a70b-5823203f84b0" (UID: "cde9335c-c895-4cd1-a70b-5823203f84b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:02.485942 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.485545    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde9335c-c895-4cd1-a70b-5823203f84b0-istiod-ca-cert" pod "cde9335c-c895-4cd1-a70b-5823203f84b0" (UID: "cde9335c-c895-4cd1-a70b-5823203f84b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:02:02.485942 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.485790    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-credential-socket" pod "cde9335c-c895-4cd1-a70b-5823203f84b0" (UID: "cde9335c-c895-4cd1-a70b-5823203f84b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:02.485942 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.485916    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-data" pod "cde9335c-c895-4cd1-a70b-5823203f84b0" (UID: "cde9335c-c895-4cd1-a70b-5823203f84b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:02.486317 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.486286    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-certs" pod "cde9335c-c895-4cd1-a70b-5823203f84b0" (UID: "cde9335c-c895-4cd1-a70b-5823203f84b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:02.488970 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.488926    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-envoy" pod "cde9335c-c895-4cd1-a70b-5823203f84b0" (UID: "cde9335c-c895-4cd1-a70b-5823203f84b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:02.489081 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.488977    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-token" pod "cde9335c-c895-4cd1-a70b-5823203f84b0" (UID: "cde9335c-c895-4cd1-a70b-5823203f84b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:02.489221 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.489193    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-podinfo" pod "cde9335c-c895-4cd1-a70b-5823203f84b0" (UID: "cde9335c-c895-4cd1-a70b-5823203f84b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:02:02.489610 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.489588    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-kube-api-access-wq9zl" pod "cde9335c-c895-4cd1-a70b-5823203f84b0" (UID: "cde9335c-c895-4cd1-a70b-5823203f84b0"). InnerVolumeSpecName "kube-api-access-wq9zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:02.586739 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.586693    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:02.586739 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.586733    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:02.586739 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.586748    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:02.587065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.586763    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:02.587065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.586778    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9zl\" (UniqueName: \"kubernetes.io/projected/cde9335c-c895-4cd1-a70b-5823203f84b0-kube-api-access-wq9zl\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:02.587065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.586792    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:02.587065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.586806    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde9335c-c895-4cd1-a70b-5823203f84b0-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:02.587065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.586819    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:02.587065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:02.586835    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde9335c-c895-4cd1-a70b-5823203f84b0-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.239236 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.239203    4555 generic.go:356] "Generic (PLEG): container finished" podID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerID="7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9" exitCode=0
Oct 31 00:02:03.239439 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.239255    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"
Oct 31 00:02:03.239439 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.239258    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" event={"ID":"cde9335c-c895-4cd1-a70b-5823203f84b0","Type":"ContainerDied","Data":"7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9"}
Oct 31 00:02:03.239439 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.239317    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w" event={"ID":"cde9335c-c895-4cd1-a70b-5823203f84b0","Type":"ContainerDied","Data":"f8dce0caeb5c7f9df1801772c057df2928e20f4c3b446384956fa179d82a2c88"}
Oct 31 00:02:03.239439 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.239342    4555 scope.go:122] "RemoveContainer" containerID="7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9"
Oct 31 00:02:03.249634 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.249613    4555 scope.go:122] "RemoveContainer" containerID="7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9"
Oct 31 00:02:03.249932 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:02:03.249882    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9\": container with ID starting with 7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9 not found: ID does not exist" containerID="7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9"
Oct 31 00:02:03.250040 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.249945    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9"} err="failed to get container status \"7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9\": rpc error: code = NotFound desc = could not find container \"7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9\": container with ID starting with 7854e79a35bc1d37ec5337b07213371ea76ffa72618442e859b24341447e97f9 not found: ID does not exist"
Oct 31 00:02:03.264898 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.264864    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"]
Oct 31 00:02:03.268992 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.268969    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3dcca-openshift-default-7d6885d9bf-v8c7w"]
Oct 31 00:02:03.685260 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.685229    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 31 00:02:03.707424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.700861    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88785011-298c-4593-9d9e-cc6b0c8f5297-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88785011-298c-4593-9d9e-cc6b0c8f5297-istiod-ca-cert\") pod \"88785011-298c-4593-9d9e-cc6b0c8f5297\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") "
Oct 31 00:02:03.707424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.700911    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-socket\") pod \"88785011-298c-4593-9d9e-cc6b0c8f5297\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") "
Oct 31 00:02:03.707424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.700955    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-certs\") pod \"88785011-298c-4593-9d9e-cc6b0c8f5297\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") "
Oct 31 00:02:03.707424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.700999    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-kube-api-access-8wzfl\" (UniqueName: \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-kube-api-access-8wzfl\") pod \"88785011-298c-4593-9d9e-cc6b0c8f5297\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") "
Oct 31 00:02:03.707424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.701028    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-data\") pod \"88785011-298c-4593-9d9e-cc6b0c8f5297\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") "
Oct 31 00:02:03.707424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.701057    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-token\" (UniqueName: \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-token\") pod \"88785011-298c-4593-9d9e-cc6b0c8f5297\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") "
Oct 31 00:02:03.707424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.701086    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-envoy\") pod \"88785011-298c-4593-9d9e-cc6b0c8f5297\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") "
Oct 31 00:02:03.707424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.701150    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-podinfo\") pod \"88785011-298c-4593-9d9e-cc6b0c8f5297\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") "
Oct 31 00:02:03.707424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.701291    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-credential-socket\") pod \"88785011-298c-4593-9d9e-cc6b0c8f5297\" (UID: \"88785011-298c-4593-9d9e-cc6b0c8f5297\") "
Oct 31 00:02:03.707424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.701829    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88785011-298c-4593-9d9e-cc6b0c8f5297-istiod-ca-cert" pod "88785011-298c-4593-9d9e-cc6b0c8f5297" (UID: "88785011-298c-4593-9d9e-cc6b0c8f5297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:02:03.723932 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.716902    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-data" pod "88785011-298c-4593-9d9e-cc6b0c8f5297" (UID: "88785011-298c-4593-9d9e-cc6b0c8f5297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:03.723932 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.721760    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-certs" pod "88785011-298c-4593-9d9e-cc6b0c8f5297" (UID: "88785011-298c-4593-9d9e-cc6b0c8f5297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:03.752868 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.739819    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-socket" pod "88785011-298c-4593-9d9e-cc6b0c8f5297" (UID: "88785011-298c-4593-9d9e-cc6b0c8f5297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:03.752868 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.743148    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-credential-socket" pod "88785011-298c-4593-9d9e-cc6b0c8f5297" (UID: "88785011-298c-4593-9d9e-cc6b0c8f5297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:03.758474 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.758413    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-kube-api-access-8wzfl" pod "88785011-298c-4593-9d9e-cc6b0c8f5297" (UID: "88785011-298c-4593-9d9e-cc6b0c8f5297"). InnerVolumeSpecName "kube-api-access-8wzfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:03.775671 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.775633    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-podinfo" pod "88785011-298c-4593-9d9e-cc6b0c8f5297" (UID: "88785011-298c-4593-9d9e-cc6b0c8f5297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:02:03.775831 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.775644    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-envoy" pod "88785011-298c-4593-9d9e-cc6b0c8f5297" (UID: "88785011-298c-4593-9d9e-cc6b0c8f5297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:03.777722 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.777696    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-token" pod "88785011-298c-4593-9d9e-cc6b0c8f5297" (UID: "88785011-298c-4593-9d9e-cc6b0c8f5297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:03.788541 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.788507    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 31 00:02:03.802581 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.802547    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9261021-433e-4641-927b-1893dd0d1f24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9261021-433e-4641-927b-1893dd0d1f24-istiod-ca-cert\") pod \"f9261021-433e-4641-927b-1893dd0d1f24\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") "
Oct 31 00:02:03.802784 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.802614    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-data\") pod \"f9261021-433e-4641-927b-1893dd0d1f24\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") "
Oct 31 00:02:03.802784 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.802658    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-socket\") pod \"f9261021-433e-4641-927b-1893dd0d1f24\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") "
Oct 31 00:02:03.802784 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.802701    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-envoy\") pod \"f9261021-433e-4641-927b-1893dd0d1f24\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") "
Oct 31 00:02:03.802784 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.802741    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-istio-token\" (UniqueName: \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-istio-token\") pod \"f9261021-433e-4641-927b-1893dd0d1f24\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") "
Oct 31 00:02:03.802784 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.802766    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-certs\") pod \"f9261021-433e-4641-927b-1893dd0d1f24\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") "
Oct 31 00:02:03.803081 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.802796    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-kube-api-access-nx8ql\" (UniqueName: \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-kube-api-access-nx8ql\") pod \"f9261021-433e-4641-927b-1893dd0d1f24\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") "
Oct 31 00:02:03.803081 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.802828    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9261021-433e-4641-927b-1893dd0d1f24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9261021-433e-4641-927b-1893dd0d1f24-istio-podinfo\") pod \"f9261021-433e-4641-927b-1893dd0d1f24\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") "
Oct 31 00:02:03.803081 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.802930    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-credential-socket\") pod \"f9261021-433e-4641-927b-1893dd0d1f24\" (UID: \"f9261021-433e-4641-927b-1893dd0d1f24\") "
Oct 31 00:02:03.803081 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.802979    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-data" pod "f9261021-433e-4641-927b-1893dd0d1f24" (UID: "f9261021-433e-4641-927b-1893dd0d1f24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:03.803081 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803017    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-socket" pod "f9261021-433e-4641-927b-1893dd0d1f24" (UID: "f9261021-433e-4641-927b-1893dd0d1f24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:03.803363 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803094    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9261021-433e-4641-927b-1893dd0d1f24-istiod-ca-cert" pod "f9261021-433e-4641-927b-1893dd0d1f24" (UID: "f9261021-433e-4641-927b-1893dd0d1f24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:02:03.803363 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803292    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-credential-socket" pod "f9261021-433e-4641-927b-1893dd0d1f24" (UID: "f9261021-433e-4641-927b-1893dd0d1f24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:03.803363 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803355    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803363 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803390    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88785011-298c-4593-9d9e-cc6b0c8f5297-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803409    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803408    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-certs" pod "f9261021-433e-4641-927b-1893dd0d1f24" (UID: "f9261021-433e-4641-927b-1893dd0d1f24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803424    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803438    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803447    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wzfl\" (UniqueName: \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-kube-api-access-8wzfl\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803456    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803464    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803474    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803484    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88785011-298c-4593-9d9e-cc6b0c8f5297-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803492    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88785011-298c-4593-9d9e-cc6b0c8f5297-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.803588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.803500    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9261021-433e-4641-927b-1893dd0d1f24-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.805285 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.805259    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-kube-api-access-nx8ql" pod "f9261021-433e-4641-927b-1893dd0d1f24" (UID: "f9261021-433e-4641-927b-1893dd0d1f24"). InnerVolumeSpecName "kube-api-access-nx8ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:03.805504 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.805350    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9261021-433e-4641-927b-1893dd0d1f24-istio-podinfo" pod "f9261021-433e-4641-927b-1893dd0d1f24" (UID: "f9261021-433e-4641-927b-1893dd0d1f24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:02:03.805837 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.805806    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-istio-token" pod "f9261021-433e-4641-927b-1893dd0d1f24" (UID: "f9261021-433e-4641-927b-1893dd0d1f24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:03.806262 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.806245    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-envoy" pod "f9261021-433e-4641-927b-1893dd0d1f24" (UID: "f9261021-433e-4641-927b-1893dd0d1f24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:03.905117 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.905071    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.905117 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.905112    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.905117 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.905127    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.905488 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.905141    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9261021-433e-4641-927b-1893dd0d1f24-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.905488 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.905157    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx8ql\" (UniqueName: \"kubernetes.io/projected/f9261021-433e-4641-927b-1893dd0d1f24-kube-api-access-nx8ql\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:03.905488 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:03.905173    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9261021-433e-4641-927b-1893dd0d1f24-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:04.024437 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.024403    4555 scope.go:122] "RemoveContainer" containerID="2770ff4b2d3daafe20a43ff4ea08e170c829741f91ed55ec5a04d3c186241ebb"
Oct 31 00:02:04.034640 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.034616    4555 scope.go:122] "RemoveContainer" containerID="7de9401a7a9db0f0a196272a9934fbcfff026a9634e8efe1fb7da4ebc282651a"
Oct 31 00:02:04.044613 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.044583    4555 scope.go:122] "RemoveContainer" containerID="a08bf4d2726289398ed7811891da48bc04db7c7f59954a41ec808539d9607f9b"
Oct 31 00:02:04.054776 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.054750    4555 scope.go:122] "RemoveContainer" containerID="aafd2075a3546fc21990cd7846023d48f4782c9384a3f11262689bc69cbf90ef"
Oct 31 00:02:04.245545 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.245498    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" event={"ID":"f9261021-433e-4641-927b-1893dd0d1f24","Type":"ContainerDied","Data":"aafd2075a3546fc21990cd7846023d48f4782c9384a3f11262689bc69cbf90ef"}
Oct 31 00:02:04.245721 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.245572    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk" event={"ID":"f9261021-433e-4641-927b-1893dd0d1f24","Type":"ContainerDied","Data":"9d10673592d4f440a00e37c82536f4a2e99bdc773f2f0ac9c93d4762e369f674"}
Oct 31 00:02:04.245721 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.245581    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"
Oct 31 00:02:04.245721 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.245581    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"
Oct 31 00:02:04.245721 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.245599    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" event={"ID":"88785011-298c-4593-9d9e-cc6b0c8f5297","Type":"ContainerDied","Data":"7de9401a7a9db0f0a196272a9934fbcfff026a9634e8efe1fb7da4ebc282651a"}
Oct 31 00:02:04.245721 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.245647    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m" event={"ID":"88785011-298c-4593-9d9e-cc6b0c8f5297","Type":"ContainerDied","Data":"0dddf4b881e14d66d0a7d85f11c22a3092d379f8f13302278b64d9e7fdf56a50"}
Oct 31 00:02:04.285926 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.285843    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"]
Oct 31 00:02:04.295198 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.295168    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-4865m"]
Oct 31 00:02:04.335883 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.335844    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"]
Oct 31 00:02:04.362029 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:04.361995    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ef03a-openshift-default-78cc76c9cc-xm6zk"]
Oct 31 00:02:05.207605 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:05.207572    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" path="/var/lib/kubelet/pods/88785011-298c-4593-9d9e-cc6b0c8f5297/volumes"
Oct 31 00:02:05.208087 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:05.208070    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" path="/var/lib/kubelet/pods/cde9335c-c895-4cd1-a70b-5823203f84b0/volumes"
Oct 31 00:02:05.208544 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:05.208530    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9261021-433e-4641-927b-1893dd0d1f24" path="/var/lib/kubelet/pods/f9261021-433e-4641-927b-1893dd0d1f24/volumes"
Oct 31 00:02:13.703605 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:13.703565    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"]
Oct 31 00:02:17.467890 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:17.467838    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:02:24.751029 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:24.750983    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"]
Oct 31 00:02:26.151666 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:26.151628    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:02:27.447233 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:02:27.447188    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:02:28.736144 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:28.736102    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn" podUID="847ba777-6164-4b2d-ac74-a90ba8553acc" containerName="main" containerID="cri-o://ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27" gracePeriod=45
Oct 31 00:02:29.002422 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.002334    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 31 00:02:29.186096 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186060    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-dshm\") pod \"847ba777-6164-4b2d-ac74-a90ba8553acc\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") "
Oct 31 00:02:29.186096 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186111    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-model-cache\") pod \"847ba777-6164-4b2d-ac74-a90ba8553acc\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") "
Oct 31 00:02:29.186411 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186142    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-tmp-dir\") pod \"847ba777-6164-4b2d-ac74-a90ba8553acc\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") "
Oct 31 00:02:29.186411 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186197    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847ba777-6164-4b2d-ac74-a90ba8553acc-kube-api-access-xxd9b\" (UniqueName: \"kubernetes.io/projected/847ba777-6164-4b2d-ac74-a90ba8553acc-kube-api-access-xxd9b\") pod \"847ba777-6164-4b2d-ac74-a90ba8553acc\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") "
Oct 31 00:02:29.186411 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186271    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847ba777-6164-4b2d-ac74-a90ba8553acc-tls-certs\" (UniqueName: \"kubernetes.io/secret/847ba777-6164-4b2d-ac74-a90ba8553acc-tls-certs\") pod \"847ba777-6164-4b2d-ac74-a90ba8553acc\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") "
Oct 31 00:02:29.186411 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186311    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-home\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-home\") pod \"847ba777-6164-4b2d-ac74-a90ba8553acc\" (UID: \"847ba777-6164-4b2d-ac74-a90ba8553acc\") "
Oct 31 00:02:29.186707 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186464    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-model-cache" pod "847ba777-6164-4b2d-ac74-a90ba8553acc" (UID: "847ba777-6164-4b2d-ac74-a90ba8553acc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:29.186707 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186512    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-tmp-dir" pod "847ba777-6164-4b2d-ac74-a90ba8553acc" (UID: "847ba777-6164-4b2d-ac74-a90ba8553acc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:29.186821 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186703    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-home" pod "847ba777-6164-4b2d-ac74-a90ba8553acc" (UID: "847ba777-6164-4b2d-ac74-a90ba8553acc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:29.186961 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186942    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:29.187039 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186968    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:29.187039 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.186985    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:29.188880 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.188850    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847ba777-6164-4b2d-ac74-a90ba8553acc-tls-certs" pod "847ba777-6164-4b2d-ac74-a90ba8553acc" (UID: "847ba777-6164-4b2d-ac74-a90ba8553acc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:02:29.189041 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.189021    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847ba777-6164-4b2d-ac74-a90ba8553acc-kube-api-access-xxd9b" pod "847ba777-6164-4b2d-ac74-a90ba8553acc" (UID: "847ba777-6164-4b2d-ac74-a90ba8553acc"). InnerVolumeSpecName "kube-api-access-xxd9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:29.189141 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.189123    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-dshm" pod "847ba777-6164-4b2d-ac74-a90ba8553acc" (UID: "847ba777-6164-4b2d-ac74-a90ba8553acc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:29.288402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.288333    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ba777-6164-4b2d-ac74-a90ba8553acc-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:29.288402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.288385    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxd9b\" (UniqueName: \"kubernetes.io/projected/847ba777-6164-4b2d-ac74-a90ba8553acc-kube-api-access-xxd9b\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:29.288402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.288402    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ba777-6164-4b2d-ac74-a90ba8553acc-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:29.433571 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.433534    4555 generic.go:356] "Generic (PLEG): container finished" podID="847ba777-6164-4b2d-ac74-a90ba8553acc" containerID="ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27" exitCode=0
Oct 31 00:02:29.433748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.433590    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"
Oct 31 00:02:29.433748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.433620    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn" event={"ID":"847ba777-6164-4b2d-ac74-a90ba8553acc","Type":"ContainerDied","Data":"ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27"}
Oct 31 00:02:29.433748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.433664    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn" event={"ID":"847ba777-6164-4b2d-ac74-a90ba8553acc","Type":"ContainerDied","Data":"5c753db5f663db1c4cc02d01f6682aacb5316c52528030d1d6245fe4b1b53b10"}
Oct 31 00:02:29.433748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.433684    4555 scope.go:122] "RemoveContainer" containerID="ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27"
Oct 31 00:02:29.443548 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.443510    4555 scope.go:122] "RemoveContainer" containerID="ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27"
Oct 31 00:02:29.443840 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:02:29.443814    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27\": container with ID starting with ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27 not found: ID does not exist" containerID="ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27"
Oct 31 00:02:29.443902 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.443846    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27"} err="failed to get container status \"ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27\": rpc error: code = NotFound desc = could not find container \"ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27\": container with ID starting with ac3412af5c1b8205b761bedb121b2a5b7a16b11c5139372847fe167eb3788d27 not found: ID does not exist"
Oct 31 00:02:29.470059 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.470025    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"]
Oct 31 00:02:29.504798 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:29.504766    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6b38a4/test-model-6b38a4-kserve-ccfdc56dc-m6vrn"]
Oct 31 00:02:31.208607 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:31.208564    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847ba777-6164-4b2d-ac74-a90ba8553acc" path="/var/lib/kubelet/pods/847ba777-6164-4b2d-ac74-a90ba8553acc/volumes"
Oct 31 00:02:32.024182 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:32.024139    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"]
Oct 31 00:02:32.026655 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:32.026630    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c9c9fc-46mp9"]
Oct 31 00:02:33.206261 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:33.206229    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ec3407-e40d-4e3e-8777-737251989527" path="/var/lib/kubelet/pods/04ec3407-e40d-4e3e-8777-737251989527/volumes"
Oct 31 00:02:35.058018 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:35.057984    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"]
Oct 31 00:02:36.662088 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:36.662047    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"]
Oct 31 00:02:36.662625 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:36.662324    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628" podUID="ab5a1218-e055-4dca-9ee2-9285fcfd1fd5" containerName="maas-api" containerID="cri-o://c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd" gracePeriod=30
Oct 31 00:02:36.970702 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:36.970677    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 31 00:02:37.057516 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.057478    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-maas-api-tls\") pod \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") "
Oct 31 00:02:37.057726 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.057594    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-openshift-service-ca\") pod \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") "
Oct 31 00:02:37.057726 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.057689    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-kube-api-access-cfcww\" (UniqueName: \"kubernetes.io/projected/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-kube-api-access-cfcww\") pod \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") "
Oct 31 00:02:37.057854 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.057742    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-metrics-certs\") pod \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\" (UID: \"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5\") "
Oct 31 00:02:37.058018 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.057993    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-openshift-service-ca" pod "ab5a1218-e055-4dca-9ee2-9285fcfd1fd5" (UID: "ab5a1218-e055-4dca-9ee2-9285fcfd1fd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:02:37.058121 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.058104    4555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-openshift-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:37.059894 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.059858    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-maas-api-tls" pod "ab5a1218-e055-4dca-9ee2-9285fcfd1fd5" (UID: "ab5a1218-e055-4dca-9ee2-9285fcfd1fd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:02:37.060030 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.059898    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-kube-api-access-cfcww" pod "ab5a1218-e055-4dca-9ee2-9285fcfd1fd5" (UID: "ab5a1218-e055-4dca-9ee2-9285fcfd1fd5"). InnerVolumeSpecName "kube-api-access-cfcww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:37.060030 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.060001    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-metrics-certs" pod "ab5a1218-e055-4dca-9ee2-9285fcfd1fd5" (UID: "ab5a1218-e055-4dca-9ee2-9285fcfd1fd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:02:37.159180 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.159143    4555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-metrics-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:37.159180 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.159172    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-maas-api-tls\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:37.159180 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.159183    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfcww\" (UniqueName: \"kubernetes.io/projected/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5-kube-api-access-cfcww\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:37.489146 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.489054    4555 generic.go:356] "Generic (PLEG): container finished" podID="ab5a1218-e055-4dca-9ee2-9285fcfd1fd5" containerID="c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd" exitCode=0
Oct 31 00:02:37.489146 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.489107    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"
Oct 31 00:02:37.489359 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.489146    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628" event={"ID":"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5","Type":"ContainerDied","Data":"c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd"}
Oct 31 00:02:37.489359 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.489204    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628" event={"ID":"ab5a1218-e055-4dca-9ee2-9285fcfd1fd5","Type":"ContainerDied","Data":"dfc52574c3f994b8d22b837632b78a7b33c72bd7a54a48f6451d921abf917d7f"}
Oct 31 00:02:37.489359 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.489233    4555 scope.go:122] "RemoveContainer" containerID="c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd"
Oct 31 00:02:37.499523 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.499505    4555 scope.go:122] "RemoveContainer" containerID="c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd"
Oct 31 00:02:37.499800 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:02:37.499782    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd\": container with ID starting with c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd not found: ID does not exist" containerID="c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd"
Oct 31 00:02:37.499875 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.499812    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd"} err="failed to get container status \"c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd\": rpc error: code = NotFound desc = could not find container \"c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd\": container with ID starting with c73990e3f53548d85a95071d11712ea708232c1e31dfb3ccdbd37ce0ae983fcd not found: ID does not exist"
Oct 31 00:02:37.514833 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.514810    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"]
Oct 31 00:02:37.520481 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:37.520457    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b38a4-65796d5f4-wk628"]
Oct 31 00:02:39.206601 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:39.206562    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5a1218-e055-4dca-9ee2-9285fcfd1fd5" path="/var/lib/kubelet/pods/ab5a1218-e055-4dca-9ee2-9285fcfd1fd5/volumes"
Oct 31 00:02:39.237427 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:39.237390    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:02:39.774808 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:39.774757    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v" podUID="153b8fe9-6fb0-4cdf-bf77-97b44eb644af" containerName="main" containerID="cri-o://453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633" gracePeriod=45
Oct 31 00:02:40.039206 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.039151    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 31 00:02:40.059035 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.058991    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp" podUID="2c8c7cd4-5bea-4777-ba9d-868d9096648d" containerName="payload-processing" containerID="cri-o://7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237" gracePeriod=25
Oct 31 00:02:40.089044 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089016    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tmp-dir\") pod \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") "
Oct 31 00:02:40.089244 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089064    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-dshm\") pod \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") "
Oct 31 00:02:40.089244 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089094    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tls-certs\" (UniqueName: \"kubernetes.io/secret/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tls-certs\") pod \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") "
Oct 31 00:02:40.089244 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089123    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-model-cache\") pod \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") "
Oct 31 00:02:40.089244 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089170    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-home\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-home\") pod \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") "
Oct 31 00:02:40.089244 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089232    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-kube-api-access-mcx7d\" (UniqueName: \"kubernetes.io/projected/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-kube-api-access-mcx7d\") pod \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\" (UID: \"153b8fe9-6fb0-4cdf-bf77-97b44eb644af\") "
Oct 31 00:02:40.089530 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089348    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tmp-dir" pod "153b8fe9-6fb0-4cdf-bf77-97b44eb644af" (UID: "153b8fe9-6fb0-4cdf-bf77-97b44eb644af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:40.089588 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089546    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-home" pod "153b8fe9-6fb0-4cdf-bf77-97b44eb644af" (UID: "153b8fe9-6fb0-4cdf-bf77-97b44eb644af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:40.089748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089722    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:40.089748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089746    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:40.090001 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.089976    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-model-cache" pod "153b8fe9-6fb0-4cdf-bf77-97b44eb644af" (UID: "153b8fe9-6fb0-4cdf-bf77-97b44eb644af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:40.092592 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.092530    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-dshm" pod "153b8fe9-6fb0-4cdf-bf77-97b44eb644af" (UID: "153b8fe9-6fb0-4cdf-bf77-97b44eb644af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:40.092592 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.092534    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tls-certs" pod "153b8fe9-6fb0-4cdf-bf77-97b44eb644af" (UID: "153b8fe9-6fb0-4cdf-bf77-97b44eb644af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:02:40.093135 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.093109    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-kube-api-access-mcx7d" pod "153b8fe9-6fb0-4cdf-bf77-97b44eb644af" (UID: "153b8fe9-6fb0-4cdf-bf77-97b44eb644af"). InnerVolumeSpecName "kube-api-access-mcx7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:40.190759 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.190720    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:40.190759 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.190748    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:40.190759 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.190757    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:40.190759 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.190769    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcx7d\" (UniqueName: \"kubernetes.io/projected/153b8fe9-6fb0-4cdf-bf77-97b44eb644af-kube-api-access-mcx7d\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:40.513017 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.512918    4555 generic.go:356] "Generic (PLEG): container finished" podID="153b8fe9-6fb0-4cdf-bf77-97b44eb644af" containerID="453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633" exitCode=0
Oct 31 00:02:40.513017 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.512970    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"
Oct 31 00:02:40.513569 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.513008    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v" event={"ID":"153b8fe9-6fb0-4cdf-bf77-97b44eb644af","Type":"ContainerDied","Data":"453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633"}
Oct 31 00:02:40.513569 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.513060    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v" event={"ID":"153b8fe9-6fb0-4cdf-bf77-97b44eb644af","Type":"ContainerDied","Data":"a9f667d91a785e5686466e1c577e8a4f10e979ac1dc7e2bcd4453b674c010ae6"}
Oct 31 00:02:40.513569 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.513082    4555 scope.go:122] "RemoveContainer" containerID="453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633"
Oct 31 00:02:40.526973 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.526955    4555 scope.go:122] "RemoveContainer" containerID="453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633"
Oct 31 00:02:40.527266 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:02:40.527243    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633\": container with ID starting with 453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633 not found: ID does not exist" containerID="453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633"
Oct 31 00:02:40.527323 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.527275    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633"} err="failed to get container status \"453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633\": rpc error: code = NotFound desc = could not find container \"453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633\": container with ID starting with 453853f0010252995aea5fdd1624cbffab388e7601ac0c84a68ec0dfc7c8e633 not found: ID does not exist"
Oct 31 00:02:40.538794 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.538771    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"]
Oct 31 00:02:40.542828 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:40.542804    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e5934f/test-model-e5934f-kserve-cf44c795-tv64v"]
Oct 31 00:02:41.001063 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:41.001021    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp" podUID="2c8c7cd4-5bea-4777-ba9d-868d9096648d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:02:41.205867 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:41.205826    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153b8fe9-6fb0-4cdf-bf77-97b44eb644af" path="/var/lib/kubelet/pods/153b8fe9-6fb0-4cdf-bf77-97b44eb644af/volumes"
Oct 31 00:02:44.838990 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:44.838955    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"]
Oct 31 00:02:44.839549 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:44.839169    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy" containerID="cri-o://04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f" gracePeriod=30
Oct 31 00:02:49.154106 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:49.154063    4555 patch_prober.go:29] interesting pod/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:02:49.154521 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:49.154139    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:02:50.102254 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.102220    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 31 00:02:50.182734 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.182655    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-socket\") pod \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") "
Oct 31 00:02:50.183181 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.182762    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-kube-api-access-g4h76\" (UniqueName: \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-kube-api-access-g4h76\") pod \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") "
Oct 31 00:02:50.183181 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.182813    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-envoy\") pod \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") "
Oct 31 00:02:50.183181 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.182862    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-certs\") pod \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") "
Oct 31 00:02:50.183181 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.182900    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-podinfo\") pod \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") "
Oct 31 00:02:50.183181 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.182937    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-token\" (UniqueName: \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-token\") pod \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") "
Oct 31 00:02:50.183181 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.182959    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-socket" pod "fdc37fbe-1eb2-43f1-aadd-2a765ce337de" (UID: "fdc37fbe-1eb2-43f1-aadd-2a765ce337de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:50.183181 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.182971    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istiod-ca-cert\") pod \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") "
Oct 31 00:02:50.183181 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.183000    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-credential-socket\") pod \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") "
Oct 31 00:02:50.183181 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.183028    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-data\") pod \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\" (UID: \"fdc37fbe-1eb2-43f1-aadd-2a765ce337de\") "
Oct 31 00:02:50.183903 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.183270    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-certs" pod "fdc37fbe-1eb2-43f1-aadd-2a765ce337de" (UID: "fdc37fbe-1eb2-43f1-aadd-2a765ce337de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:50.183903 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.183281    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-credential-socket" pod "fdc37fbe-1eb2-43f1-aadd-2a765ce337de" (UID: "fdc37fbe-1eb2-43f1-aadd-2a765ce337de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:50.183903 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.183353    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istiod-ca-cert" pod "fdc37fbe-1eb2-43f1-aadd-2a765ce337de" (UID: "fdc37fbe-1eb2-43f1-aadd-2a765ce337de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:02:50.183903 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.183454    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:50.183903 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.183474    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:50.183903 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.183489    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:50.183903 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.183507    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-data" pod "fdc37fbe-1eb2-43f1-aadd-2a765ce337de" (UID: "fdc37fbe-1eb2-43f1-aadd-2a765ce337de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:50.185355 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.185329    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-envoy" pod "fdc37fbe-1eb2-43f1-aadd-2a765ce337de" (UID: "fdc37fbe-1eb2-43f1-aadd-2a765ce337de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:02:50.185487 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.185413    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-token" pod "fdc37fbe-1eb2-43f1-aadd-2a765ce337de" (UID: "fdc37fbe-1eb2-43f1-aadd-2a765ce337de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:50.185625 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.185601    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-podinfo" pod "fdc37fbe-1eb2-43f1-aadd-2a765ce337de" (UID: "fdc37fbe-1eb2-43f1-aadd-2a765ce337de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:02:50.185625 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.185613    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-kube-api-access-g4h76" pod "fdc37fbe-1eb2-43f1-aadd-2a765ce337de" (UID: "fdc37fbe-1eb2-43f1-aadd-2a765ce337de"). InnerVolumeSpecName "kube-api-access-g4h76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:50.284198 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.284165    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4h76\" (UniqueName: \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-kube-api-access-g4h76\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:50.284198 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.284196    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:50.284198 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.284206    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:50.284476 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.284215    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:50.284476 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.284223    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:50.284476 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.284231    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc37fbe-1eb2-43f1-aadd-2a765ce337de-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:50.578492 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.578457    4555 generic.go:356] "Generic (PLEG): container finished" podID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerID="04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f" exitCode=0
Oct 31 00:02:50.578708 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.578504    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"
Oct 31 00:02:50.578708 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.578524    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" event={"ID":"fdc37fbe-1eb2-43f1-aadd-2a765ce337de","Type":"ContainerDied","Data":"04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f"}
Oct 31 00:02:50.578708 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.578577    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc" event={"ID":"fdc37fbe-1eb2-43f1-aadd-2a765ce337de","Type":"ContainerDied","Data":"0eaa5635e4ca5dff66090fa7331614806fb7025c053428038100710732117c41"}
Oct 31 00:02:50.578708 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.578603    4555 scope.go:122] "RemoveContainer" containerID="04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f"
Oct 31 00:02:50.589628 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.589612    4555 scope.go:122] "RemoveContainer" containerID="04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f"
Oct 31 00:02:50.589905 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:02:50.589887    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f\": container with ID starting with 04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f not found: ID does not exist" containerID="04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f"
Oct 31 00:02:50.589950 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.589913    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f"} err="failed to get container status \"04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f\": rpc error: code = NotFound desc = could not find container \"04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f\": container with ID starting with 04623b1171ea6c0aa809999d7ede2a0ea53b247ae821b38343e69636935d0b9f not found: ID does not exist"
Oct 31 00:02:50.604327 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.604300    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"]
Oct 31 00:02:50.614404 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:50.614352    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b38a4-openshift-default-5569579f7f-m6bkc"]
Oct 31 00:02:51.000593 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:51.000495    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp" podUID="2c8c7cd4-5bea-4777-ba9d-868d9096648d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:02:51.207128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:51.207095    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" path="/var/lib/kubelet/pods/fdc37fbe-1eb2-43f1-aadd-2a765ce337de/volumes"
Oct 31 00:02:56.820098 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:56.820057    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"]
Oct 31 00:02:56.820644 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:56.820358    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w" podUID="ca96124c-888e-4d50-b285-ef517527688a" containerName="maas-api" containerID="cri-o://79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064" gracePeriod=30
Oct 31 00:02:57.098197 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.098173    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 31 00:02:57.157542 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.157508    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca96124c-888e-4d50-b285-ef517527688a-kube-api-access-7b9vj\" (UniqueName: \"kubernetes.io/projected/ca96124c-888e-4d50-b285-ef517527688a-kube-api-access-7b9vj\") pod \"ca96124c-888e-4d50-b285-ef517527688a\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") "
Oct 31 00:02:57.157732 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.157596    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca96124c-888e-4d50-b285-ef517527688a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca96124c-888e-4d50-b285-ef517527688a-openshift-service-ca\") pod \"ca96124c-888e-4d50-b285-ef517527688a\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") "
Oct 31 00:02:57.157732 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.157664    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-metrics-certs\") pod \"ca96124c-888e-4d50-b285-ef517527688a\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") "
Oct 31 00:02:57.157732 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.157697    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-maas-api-tls\") pod \"ca96124c-888e-4d50-b285-ef517527688a\" (UID: \"ca96124c-888e-4d50-b285-ef517527688a\") "
Oct 31 00:02:57.157994 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.157971    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca96124c-888e-4d50-b285-ef517527688a-openshift-service-ca" pod "ca96124c-888e-4d50-b285-ef517527688a" (UID: "ca96124c-888e-4d50-b285-ef517527688a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:02:57.159920 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.159893    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-metrics-certs" pod "ca96124c-888e-4d50-b285-ef517527688a" (UID: "ca96124c-888e-4d50-b285-ef517527688a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:02:57.160045 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.159925    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca96124c-888e-4d50-b285-ef517527688a-kube-api-access-7b9vj" pod "ca96124c-888e-4d50-b285-ef517527688a" (UID: "ca96124c-888e-4d50-b285-ef517527688a"). InnerVolumeSpecName "kube-api-access-7b9vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:57.160045 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.160029    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-maas-api-tls" pod "ca96124c-888e-4d50-b285-ef517527688a" (UID: "ca96124c-888e-4d50-b285-ef517527688a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:02:57.258806 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.258771    4555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-metrics-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:57.258806 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.258802    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca96124c-888e-4d50-b285-ef517527688a-maas-api-tls\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:57.259023 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.258815    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b9vj\" (UniqueName: \"kubernetes.io/projected/ca96124c-888e-4d50-b285-ef517527688a-kube-api-access-7b9vj\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:57.259023 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.258825    4555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca96124c-888e-4d50-b285-ef517527688a-openshift-service-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:57.445246 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:02:57.445160    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:02:57.623398 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.623351    4555 generic.go:356] "Generic (PLEG): container finished" podID="ca96124c-888e-4d50-b285-ef517527688a" containerID="79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064" exitCode=0
Oct 31 00:02:57.623620 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.623422    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"
Oct 31 00:02:57.623620 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.623442    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w" event={"ID":"ca96124c-888e-4d50-b285-ef517527688a","Type":"ContainerDied","Data":"79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064"}
Oct 31 00:02:57.623620 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.623507    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w" event={"ID":"ca96124c-888e-4d50-b285-ef517527688a","Type":"ContainerDied","Data":"434e6b2e50412ab082e79e409ff3236221ec0d8d54bb7ba990136e5e465c3b9d"}
Oct 31 00:02:57.623620 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.623540    4555 scope.go:122] "RemoveContainer" containerID="79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064"
Oct 31 00:02:57.634026 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.634004    4555 scope.go:122] "RemoveContainer" containerID="79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064"
Oct 31 00:02:57.634298 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:02:57.634280    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064\": container with ID starting with 79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064 not found: ID does not exist" containerID="79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064"
Oct 31 00:02:57.634393 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.634305    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064"} err="failed to get container status \"79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064\": rpc error: code = NotFound desc = could not find container \"79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064\": container with ID starting with 79968e28c6d6d08678210d9142d8e6beae9dc492d1711d142b98f4d9d8968064 not found: ID does not exist"
Oct 31 00:02:57.644600 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.644575    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"]
Oct 31 00:02:57.647983 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:57.647959    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5934f-74455b7c5-bd28w"]
Oct 31 00:02:59.208703 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:59.208665    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca96124c-888e-4d50-b285-ef517527688a" path="/var/lib/kubelet/pods/ca96124c-888e-4d50-b285-ef517527688a/volumes"
Oct 31 00:02:59.358660 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:02:59.358625    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:03:00.327735 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.327673    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 31 00:03:00.391191 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.391143    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8c7cd4-5bea-4777-ba9d-868d9096648d-kube-api-access-qmbqk\" (UniqueName: \"kubernetes.io/projected/2c8c7cd4-5bea-4777-ba9d-868d9096648d-kube-api-access-qmbqk\") pod \"2c8c7cd4-5bea-4777-ba9d-868d9096648d\" (UID: \"2c8c7cd4-5bea-4777-ba9d-868d9096648d\") "
Oct 31 00:03:00.391425 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.391252    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8c7cd4-5bea-4777-ba9d-868d9096648d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8c7cd4-5bea-4777-ba9d-868d9096648d-plugins-config-volume\") pod \"2c8c7cd4-5bea-4777-ba9d-868d9096648d\" (UID: \"2c8c7cd4-5bea-4777-ba9d-868d9096648d\") "
Oct 31 00:03:00.391683 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.391654    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8c7cd4-5bea-4777-ba9d-868d9096648d-plugins-config-volume" pod "2c8c7cd4-5bea-4777-ba9d-868d9096648d" (UID: "2c8c7cd4-5bea-4777-ba9d-868d9096648d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:00.394022 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.394002    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8c7cd4-5bea-4777-ba9d-868d9096648d-kube-api-access-qmbqk" pod "2c8c7cd4-5bea-4777-ba9d-868d9096648d" (UID: "2c8c7cd4-5bea-4777-ba9d-868d9096648d"). InnerVolumeSpecName "kube-api-access-qmbqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:00.492189 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.492157    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmbqk\" (UniqueName: \"kubernetes.io/projected/2c8c7cd4-5bea-4777-ba9d-868d9096648d-kube-api-access-qmbqk\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:00.492340 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.492194    4555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8c7cd4-5bea-4777-ba9d-868d9096648d-plugins-config-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:00.645142 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.645047    4555 generic.go:356] "Generic (PLEG): container finished" podID="2c8c7cd4-5bea-4777-ba9d-868d9096648d" containerID="7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237" exitCode=0
Oct 31 00:03:00.645142 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.645103    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"
Oct 31 00:03:00.645142 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.645127    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp" event={"ID":"2c8c7cd4-5bea-4777-ba9d-868d9096648d","Type":"ContainerDied","Data":"7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237"}
Oct 31 00:03:00.645434 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.645192    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp" event={"ID":"2c8c7cd4-5bea-4777-ba9d-868d9096648d","Type":"ContainerDied","Data":"c8bdf7f7dc7cb6dd7793e3c472b54cd69f22da449ea2188a0a86141149fa6cd8"}
Oct 31 00:03:00.645434 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.645221    4555 scope.go:122] "RemoveContainer" containerID="7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237"
Oct 31 00:03:00.656159 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.656145    4555 scope.go:122] "RemoveContainer" containerID="7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237"
Oct 31 00:03:00.656430 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:03:00.656410    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237\": container with ID starting with 7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237 not found: ID does not exist" containerID="7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237"
Oct 31 00:03:00.656477 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.656439    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237"} err="failed to get container status \"7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237\": rpc error: code = NotFound desc = could not find container \"7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237\": container with ID starting with 7dc63398aaa22f067861d394c6bdafc92e5a2b65f2d051a3d4bd5b95b7523237 not found: ID does not exist"
Oct 31 00:03:00.671506 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.671482    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"]
Oct 31 00:03:00.673578 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:00.673559    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b38a4-5b65c7f549-nm6mp"]
Oct 31 00:03:01.208043 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:01.208007    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8c7cd4-5bea-4777-ba9d-868d9096648d" path="/var/lib/kubelet/pods/2c8c7cd4-5bea-4777-ba9d-868d9096648d/volumes"
Oct 31 00:03:04.125459 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:04.125430    4555 scope.go:122] "RemoveContainer" containerID="3f1b5818eba2bf2586500728d61c541b0dfcaebae1837daaa610d8d34ba3f457"
Oct 31 00:03:10.746597 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:10.746564    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"]
Oct 31 00:03:16.559742 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.559657    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"]
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560024    4555 cpu_manager.go:414] "containerMap: removing container" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560034    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76874aeb-95f4-483e-acca-cb1ba0b9ecfc" containerName="istio-proxy"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560050    4555 cpu_manager.go:414] "containerMap: removing container" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560055    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560063    4555 cpu_manager.go:414] "containerMap: removing container" podUID="5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6" containerName="payload-processing"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560067    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6" containerName="payload-processing"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560076    4555 cpu_manager.go:414] "containerMap: removing container" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560080    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560087    4555 cpu_manager.go:414] "containerMap: removing container" podUID="ab5a1218-e055-4dca-9ee2-9285fcfd1fd5" containerName="maas-api"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560091    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5a1218-e055-4dca-9ee2-9285fcfd1fd5" containerName="maas-api"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560101    4555 cpu_manager.go:414] "containerMap: removing container" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560105    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560115    4555 cpu_manager.go:414] "containerMap: removing container" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main"
Oct 31 00:03:16.560119 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560121    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560128    4555 cpu_manager.go:414] "containerMap: removing container" podUID="1493b5fd-8940-438b-9063-81aa1ae5a1c3" containerName="revoke-keys"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560133    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1493b5fd-8940-438b-9063-81aa1ae5a1c3" containerName="revoke-keys"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560140    4555 cpu_manager.go:414] "containerMap: removing container" podUID="153b8fe9-6fb0-4cdf-bf77-97b44eb644af" containerName="main"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560145    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153b8fe9-6fb0-4cdf-bf77-97b44eb644af" containerName="main"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560150    4555 cpu_manager.go:414] "containerMap: removing container" podUID="ca96124c-888e-4d50-b285-ef517527688a" containerName="maas-api"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560154    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca96124c-888e-4d50-b285-ef517527688a" containerName="maas-api"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560161    4555 cpu_manager.go:414] "containerMap: removing container" podUID="2c8c7cd4-5bea-4777-ba9d-868d9096648d" containerName="payload-processing"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560165    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8c7cd4-5bea-4777-ba9d-868d9096648d" containerName="payload-processing"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560173    4555 cpu_manager.go:414] "containerMap: removing container" podUID="727002c6-942f-4ea0-b31e-74acc34261d6" containerName="cleanup"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560178    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727002c6-942f-4ea0-b31e-74acc34261d6" containerName="cleanup"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560191    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerName="cleanup"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560195    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0618ce43-2b19-49bb-9dd5-6813e8755b84" containerName="cleanup"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560201    4555 cpu_manager.go:414] "containerMap: removing container" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560206    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560214    4555 cpu_manager.go:414] "containerMap: removing container" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560218    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2029c09-8186-46dd-bc4f-aab88bde8fd1" containerName="istio-proxy"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560225    4555 cpu_manager.go:414] "containerMap: removing container" podUID="847ba777-6164-4b2d-ac74-a90ba8553acc" containerName="main"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560229    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847ba777-6164-4b2d-ac74-a90ba8553acc" containerName="main"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560235    4555 cpu_manager.go:414] "containerMap: removing container" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560239    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560299    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde9335c-c895-4cd1-a70b-5823203f84b0" containerName="istio-proxy"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560308    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="847ba777-6164-4b2d-ac74-a90ba8553acc" containerName="main"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560314    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="88785011-298c-4593-9d9e-cc6b0c8f5297" containerName="istio-proxy"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560321    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc37fbe-1eb2-43f1-aadd-2a765ce337de" containerName="istio-proxy"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560326    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0e6ddc-4637-46b7-b4e6-aaec3d2182c6" containerName="payload-processing"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560332    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9261021-433e-4641-927b-1893dd0d1f24" containerName="istio-proxy"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560336    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5a1218-e055-4dca-9ee2-9285fcfd1fd5" containerName="maas-api"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560342    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="16bfafce-6391-4dc5-8eb4-2a77f0f22164" containerName="main"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560348    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddac194a-9d84-41d6-b40b-e7aa74945352" containerName="main"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560353    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="1493b5fd-8940-438b-9063-81aa1ae5a1c3" containerName="revoke-keys"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560357    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="153b8fe9-6fb0-4cdf-bf77-97b44eb644af" containerName="main"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560361    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca96124c-888e-4d50-b285-ef517527688a" containerName="maas-api"
Oct 31 00:03:16.560656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.560381    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8c7cd4-5bea-4777-ba9d-868d9096648d" containerName="payload-processing"
Oct 31 00:03:16.565412 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.565360    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:16.577333 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.577307    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"]
Oct 31 00:03:16.663856 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.663818    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:03:16.663856 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.663818    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:03:16.663856 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.663853    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 31 00:03:16.664117 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.663899    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:03:16.738489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.738457    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9cfcd8-k65dt\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:16.738703 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.738506    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9cfcd8-k65dt\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:16.738703 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.738638    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87lfd\" (UniqueName: \"kubernetes.io/projected/ae1f96f9-25ce-489e-8f6d-4788685c83a0-kube-api-access-87lfd\") pod \"maas-api-revoke-sub-2d9cfcd8-k65dt\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:16.839675 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.839585    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9cfcd8-k65dt\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:16.839675 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.839648    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9cfcd8-k65dt\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:16.839867 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.839694    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87lfd\" (UniqueName: \"kubernetes.io/projected/ae1f96f9-25ce-489e-8f6d-4788685c83a0-kube-api-access-87lfd\") pod \"maas-api-revoke-sub-2d9cfcd8-k65dt\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:16.840200 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.840177    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9cfcd8-k65dt\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:16.842184 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.842162    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9cfcd8-k65dt\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:16.852143 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.852122    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87lfd\" (UniqueName: \"kubernetes.io/projected/ae1f96f9-25ce-489e-8f6d-4788685c83a0-kube-api-access-87lfd\") pod \"maas-api-revoke-sub-2d9cfcd8-k65dt\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:16.882518 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:16.882496    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:17.163942 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:17.163911    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"]
Oct 31 00:03:17.165532 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:03:17.165500    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1f96f9_25ce_489e_8f6d_4788685c83a0.slice/crio-88f43acea936ab0a4be8cb2e42e5269e166e54be7e3cc1b3803c130d4a4f4889 WatchSource:0}: Error finding container 88f43acea936ab0a4be8cb2e42e5269e166e54be7e3cc1b3803c130d4a4f4889: Status 404 returned error can't find the container with id 88f43acea936ab0a4be8cb2e42e5269e166e54be7e3cc1b3803c130d4a4f4889
Oct 31 00:03:17.753583 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:17.753548    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt" event={"ID":"ae1f96f9-25ce-489e-8f6d-4788685c83a0","Type":"ContainerStarted","Data":"cb1ec9085f10bc8d5e49476e303db91001ac50f0feee54ab241101bcf7949501"}
Oct 31 00:03:17.754088 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:17.753600    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt" event={"ID":"ae1f96f9-25ce-489e-8f6d-4788685c83a0","Type":"ContainerStarted","Data":"88f43acea936ab0a4be8cb2e42e5269e166e54be7e3cc1b3803c130d4a4f4889"}
Oct 31 00:03:17.774281 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:17.774221    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt" podStartSLOduration=1.774207306 podStartE2EDuration="1.774207306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:17.772359543 +0000 UTC m=+2656.698997129" watchObservedRunningTime="2026-10-31 00:03:17.774207306 +0000 UTC m=+2656.700844886"
Oct 31 00:03:18.761450 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:18.761413    4555 generic.go:356] "Generic (PLEG): container finished" podID="ae1f96f9-25ce-489e-8f6d-4788685c83a0" containerID="cb1ec9085f10bc8d5e49476e303db91001ac50f0feee54ab241101bcf7949501" exitCode=0
Oct 31 00:03:18.761450 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:18.761459    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt" event={"ID":"ae1f96f9-25ce-489e-8f6d-4788685c83a0","Type":"ContainerDied","Data":"cb1ec9085f10bc8d5e49476e303db91001ac50f0feee54ab241101bcf7949501"}
Oct 31 00:03:19.917978 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:19.917946    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:20.069504 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.069465    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-openshift-ca\") pod \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") "
Oct 31 00:03:20.069733 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.069558    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-xks-ca\") pod \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") "
Oct 31 00:03:20.069733 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.069607    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1f96f9-25ce-489e-8f6d-4788685c83a0-kube-api-access-87lfd\" (UniqueName: \"kubernetes.io/projected/ae1f96f9-25ce-489e-8f6d-4788685c83a0-kube-api-access-87lfd\") pod \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\" (UID: \"ae1f96f9-25ce-489e-8f6d-4788685c83a0\") "
Oct 31 00:03:20.069938 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.069906    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-openshift-ca" pod "ae1f96f9-25ce-489e-8f6d-4788685c83a0" (UID: "ae1f96f9-25ce-489e-8f6d-4788685c83a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:20.071969 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.071937    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1f96f9-25ce-489e-8f6d-4788685c83a0-kube-api-access-87lfd" pod "ae1f96f9-25ce-489e-8f6d-4788685c83a0" (UID: "ae1f96f9-25ce-489e-8f6d-4788685c83a0"). InnerVolumeSpecName "kube-api-access-87lfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:20.072182 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.072157    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-xks-ca" pod "ae1f96f9-25ce-489e-8f6d-4788685c83a0" (UID: "ae1f96f9-25ce-489e-8f6d-4788685c83a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:03:20.170923 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.170883    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:20.170923 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.170925    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1f96f9-25ce-489e-8f6d-4788685c83a0-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:20.171139 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.170940    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87lfd\" (UniqueName: \"kubernetes.io/projected/ae1f96f9-25ce-489e-8f6d-4788685c83a0-kube-api-access-87lfd\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:20.775175 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.775125    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt" event={"ID":"ae1f96f9-25ce-489e-8f6d-4788685c83a0","Type":"ContainerDied","Data":"88f43acea936ab0a4be8cb2e42e5269e166e54be7e3cc1b3803c130d4a4f4889"}
Oct 31 00:03:20.775175 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.775168    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"
Oct 31 00:03:20.775441 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:20.775192    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88f43acea936ab0a4be8cb2e42e5269e166e54be7e3cc1b3803c130d4a4f4889"
Oct 31 00:03:25.768274 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:25.768216    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main" containerID="cri-o://7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be" gracePeriod=45
Oct 31 00:03:26.043689 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.043663    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 31 00:03:26.125723 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.125685    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-dshm\") pod \"36867ec7-b651-4774-b248-c3f6b6696c69\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") "
Oct 31 00:03:26.125723 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.125722    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-home\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-home\") pod \"36867ec7-b651-4774-b248-c3f6b6696c69\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") "
Oct 31 00:03:26.125977 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.125845    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36867ec7-b651-4774-b248-c3f6b6696c69-tls-certs\" (UniqueName: \"kubernetes.io/secret/36867ec7-b651-4774-b248-c3f6b6696c69-tls-certs\") pod \"36867ec7-b651-4774-b248-c3f6b6696c69\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") "
Oct 31 00:03:26.125977 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.125875    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-model-cache\") pod \"36867ec7-b651-4774-b248-c3f6b6696c69\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") "
Oct 31 00:03:26.125977 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.125908    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36867ec7-b651-4774-b248-c3f6b6696c69-kube-api-access-8t8g9\" (UniqueName: \"kubernetes.io/projected/36867ec7-b651-4774-b248-c3f6b6696c69-kube-api-access-8t8g9\") pod \"36867ec7-b651-4774-b248-c3f6b6696c69\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") "
Oct 31 00:03:26.125977 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.125974    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-tmp-dir\") pod \"36867ec7-b651-4774-b248-c3f6b6696c69\" (UID: \"36867ec7-b651-4774-b248-c3f6b6696c69\") "
Oct 31 00:03:26.126186 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.126093    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-home" pod "36867ec7-b651-4774-b248-c3f6b6696c69" (UID: "36867ec7-b651-4774-b248-c3f6b6696c69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:26.126186 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.126166    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-model-cache" pod "36867ec7-b651-4774-b248-c3f6b6696c69" (UID: "36867ec7-b651-4774-b248-c3f6b6696c69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:26.126414 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.126388    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-tmp-dir" pod "36867ec7-b651-4774-b248-c3f6b6696c69" (UID: "36867ec7-b651-4774-b248-c3f6b6696c69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:26.126549 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.126455    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:26.126549 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.126480    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:26.128300 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.128270    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36867ec7-b651-4774-b248-c3f6b6696c69-kube-api-access-8t8g9" pod "36867ec7-b651-4774-b248-c3f6b6696c69" (UID: "36867ec7-b651-4774-b248-c3f6b6696c69"). InnerVolumeSpecName "kube-api-access-8t8g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:26.128595 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.128572    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36867ec7-b651-4774-b248-c3f6b6696c69-tls-certs" pod "36867ec7-b651-4774-b248-c3f6b6696c69" (UID: "36867ec7-b651-4774-b248-c3f6b6696c69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:03:26.128678 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.128575    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-dshm" pod "36867ec7-b651-4774-b248-c3f6b6696c69" (UID: "36867ec7-b651-4774-b248-c3f6b6696c69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:26.227326 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.227238    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36867ec7-b651-4774-b248-c3f6b6696c69-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:26.227326 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.227274    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t8g9\" (UniqueName: \"kubernetes.io/projected/36867ec7-b651-4774-b248-c3f6b6696c69-kube-api-access-8t8g9\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:26.227326 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.227290    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:26.227326 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.227302    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36867ec7-b651-4774-b248-c3f6b6696c69-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:26.815495 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.815462    4555 generic.go:356] "Generic (PLEG): container finished" podID="36867ec7-b651-4774-b248-c3f6b6696c69" containerID="7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be" exitCode=0
Oct 31 00:03:26.815965 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.815526    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"
Oct 31 00:03:26.815965 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.815536    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" event={"ID":"36867ec7-b651-4774-b248-c3f6b6696c69","Type":"ContainerDied","Data":"7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be"}
Oct 31 00:03:26.815965 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.815569    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj" event={"ID":"36867ec7-b651-4774-b248-c3f6b6696c69","Type":"ContainerDied","Data":"e9c37bd4fbbe45b04d979a8f62cb2a70f506a8eaef60d073c5b3d7f977c7e3cf"}
Oct 31 00:03:26.815965 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.815585    4555 scope.go:122] "RemoveContainer" containerID="7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be"
Oct 31 00:03:26.826275 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.825923    4555 scope.go:122] "RemoveContainer" containerID="7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be"
Oct 31 00:03:26.826275 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:03:26.826199    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be\": container with ID starting with 7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be not found: ID does not exist" containerID="7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be"
Oct 31 00:03:26.826275 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.826231    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be"} err="failed to get container status \"7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be\": rpc error: code = NotFound desc = could not find container \"7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be\": container with ID starting with 7977e4971840ba49b877387eb71700127befda1f2f9ad696b237b6998e9163be not found: ID does not exist"
Oct 31 00:03:26.843592 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.843566    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"]
Oct 31 00:03:26.850073 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:26.850049    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-dd3a15/simulator-w5-dd3a15-kserve-b84f58498-c7tnj"]
Oct 31 00:03:27.206179 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:27.206098    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" path="/var/lib/kubelet/pods/36867ec7-b651-4774-b248-c3f6b6696c69/volumes"
Oct 31 00:03:27.448214 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:03:27.448177    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:03:28.846630 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:28.846592    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:03:34.551098 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:34.551064    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:03:39.271914 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:39.271879    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"]
Oct 31 00:03:39.272779 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:39.272735    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy" containerID="cri-o://8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2" gracePeriod=30
Oct 31 00:03:44.556508 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.556445    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 31 00:03:44.698761 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.698718    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-podinfo\") pod \"3943f43f-9105-48e1-b60b-2ca22b65aa80\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") "
Oct 31 00:03:44.698761 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.698768    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-envoy\") pod \"3943f43f-9105-48e1-b60b-2ca22b65aa80\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") "
Oct 31 00:03:44.699032 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.698812    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-credential-socket\") pod \"3943f43f-9105-48e1-b60b-2ca22b65aa80\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") "
Oct 31 00:03:44.699032 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.698841    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-data\") pod \"3943f43f-9105-48e1-b60b-2ca22b65aa80\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") "
Oct 31 00:03:44.699032 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.698940    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-kube-api-access-ldnfl\" (UniqueName: \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-kube-api-access-ldnfl\") pod \"3943f43f-9105-48e1-b60b-2ca22b65aa80\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") "
Oct 31 00:03:44.699032 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.698971    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-certs\") pod \"3943f43f-9105-48e1-b60b-2ca22b65aa80\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") "
Oct 31 00:03:44.699032 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.699011    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-socket\") pod \"3943f43f-9105-48e1-b60b-2ca22b65aa80\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") "
Oct 31 00:03:44.699347 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.699039    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-token\" (UniqueName: \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-token\") pod \"3943f43f-9105-48e1-b60b-2ca22b65aa80\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") "
Oct 31 00:03:44.699347 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.699099    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3943f43f-9105-48e1-b60b-2ca22b65aa80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3943f43f-9105-48e1-b60b-2ca22b65aa80-istiod-ca-cert\") pod \"3943f43f-9105-48e1-b60b-2ca22b65aa80\" (UID: \"3943f43f-9105-48e1-b60b-2ca22b65aa80\") "
Oct 31 00:03:44.699555 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.699498    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-socket" pod "3943f43f-9105-48e1-b60b-2ca22b65aa80" (UID: "3943f43f-9105-48e1-b60b-2ca22b65aa80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:44.699722 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.699683    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3943f43f-9105-48e1-b60b-2ca22b65aa80-istiod-ca-cert" pod "3943f43f-9105-48e1-b60b-2ca22b65aa80" (UID: "3943f43f-9105-48e1-b60b-2ca22b65aa80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:44.699863 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.699742    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-data" pod "3943f43f-9105-48e1-b60b-2ca22b65aa80" (UID: "3943f43f-9105-48e1-b60b-2ca22b65aa80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:44.699863 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.699772    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-credential-socket" pod "3943f43f-9105-48e1-b60b-2ca22b65aa80" (UID: "3943f43f-9105-48e1-b60b-2ca22b65aa80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:44.699863 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.699800    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-certs" pod "3943f43f-9105-48e1-b60b-2ca22b65aa80" (UID: "3943f43f-9105-48e1-b60b-2ca22b65aa80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:44.702206 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.702173    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-envoy" pod "3943f43f-9105-48e1-b60b-2ca22b65aa80" (UID: "3943f43f-9105-48e1-b60b-2ca22b65aa80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:44.702340 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.702277    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-token" pod "3943f43f-9105-48e1-b60b-2ca22b65aa80" (UID: "3943f43f-9105-48e1-b60b-2ca22b65aa80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:44.702340 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.702310    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-podinfo" pod "3943f43f-9105-48e1-b60b-2ca22b65aa80" (UID: "3943f43f-9105-48e1-b60b-2ca22b65aa80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:03:44.702512 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.702391    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-kube-api-access-ldnfl" pod "3943f43f-9105-48e1-b60b-2ca22b65aa80" (UID: "3943f43f-9105-48e1-b60b-2ca22b65aa80"). InnerVolumeSpecName "kube-api-access-ldnfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:44.800883 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.800843    4555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3943f43f-9105-48e1-b60b-2ca22b65aa80-istiod-ca-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:44.800883 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.800884    4555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-podinfo\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:44.800883 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.800899    4555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-envoy\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:44.801237 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.800914    4555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-credential-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:44.801237 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.800929    4555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-data\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:44.801237 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.800951    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldnfl\" (UniqueName: \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-kube-api-access-ldnfl\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:44.801237 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.800967    4555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:44.801237 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.800981    4555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3943f43f-9105-48e1-b60b-2ca22b65aa80-workload-socket\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:44.801237 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.800995    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3943f43f-9105-48e1-b60b-2ca22b65aa80-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:44.939651 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.939567    4555 generic.go:356] "Generic (PLEG): container finished" podID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerID="8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2" exitCode=0
Oct 31 00:03:44.939651 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.939610    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" event={"ID":"3943f43f-9105-48e1-b60b-2ca22b65aa80","Type":"ContainerDied","Data":"8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2"}
Oct 31 00:03:44.939651 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.939623    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"
Oct 31 00:03:44.939651 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.939653    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s" event={"ID":"3943f43f-9105-48e1-b60b-2ca22b65aa80","Type":"ContainerDied","Data":"00318b5b48dfb78d8cd467c60de21002497d65879548d215c5ff03a339b4e0a7"}
Oct 31 00:03:44.939989 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.939674    4555 scope.go:122] "RemoveContainer" containerID="8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2"
Oct 31 00:03:44.950643 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.950622    4555 scope.go:122] "RemoveContainer" containerID="8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2"
Oct 31 00:03:44.950915 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:03:44.950893    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2\": container with ID starting with 8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2 not found: ID does not exist" containerID="8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2"
Oct 31 00:03:44.950972 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.950924    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2"} err="failed to get container status \"8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2\": rpc error: code = NotFound desc = could not find container \"8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2\": container with ID starting with 8db78f361e11d9acab850826782ff036bc1e96d738475c95fb1598539881c3a2 not found: ID does not exist"
Oct 31 00:03:44.963769 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.963746    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"]
Oct 31 00:03:44.969223 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:44.969193    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dd3a15-openshift-default-88dc7c85f-qkm6s"]
Oct 31 00:03:45.206472 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:45.206395    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" path="/var/lib/kubelet/pods/3943f43f-9105-48e1-b60b-2ca22b65aa80/volumes"
Oct 31 00:03:53.036589 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:53.036552    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"]
Oct 31 00:03:53.041102 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:53.041074    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f993bcbd-tmckl"]
Oct 31 00:03:53.205356 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:53.205303    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf68bc5-624f-47d6-b182-852ef9f76756" path="/var/lib/kubelet/pods/bcf68bc5-624f-47d6-b182-852ef9f76756/volumes"
Oct 31 00:03:54.023809 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:54.023758    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"]
Oct 31 00:03:54.025835 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:54.025807    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88093177-h2c42"]
Oct 31 00:03:55.205866 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:03:55.205830    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda8f2b1-c6f2-49a4-b7c3-ae7c096885da" path="/var/lib/kubelet/pods/bda8f2b1-c6f2-49a4-b7c3-ae7c096885da/volumes"
Oct 31 00:03:57.445238 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:03:57.445198    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:04:04.188151 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:04.188119    4555 scope.go:122] "RemoveContainer" containerID="cdc9ffa2b0e1b7ee79d0f19288f0f59e21ef1619df4ebba05a82461f6b459f49"
Oct 31 00:04:04.197656 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:04.197637    4555 scope.go:122] "RemoveContainer" containerID="d48f9a5fc140ca0c14d9dab9d3fa2492f8b38c5c47bd5fa1cdff3e58d6961e65"
Oct 31 00:04:04.206789 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:04.206761    4555 scope.go:122] "RemoveContainer" containerID="b289419cf4f2aefb6b337120595c377195d1d8651e7b93a76a2947ebfaf50f0f"
Oct 31 00:04:04.214960 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:04.214935    4555 scope.go:122] "RemoveContainer" containerID="8400a2ec920336c1b490581dbffc18325978eb06343dd4320e28b0f47086dcba"
Oct 31 00:04:10.988224 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.988185    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"]
Oct 31 00:04:10.988691 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.988585    4555 cpu_manager.go:414] "containerMap: removing container" podUID="ae1f96f9-25ce-489e-8f6d-4788685c83a0" containerName="revoke-keys"
Oct 31 00:04:10.988691 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.988599    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1f96f9-25ce-489e-8f6d-4788685c83a0" containerName="revoke-keys"
Oct 31 00:04:10.988691 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.988623    4555 cpu_manager.go:414] "containerMap: removing container" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy"
Oct 31 00:04:10.988691 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.988629    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy"
Oct 31 00:04:10.988691 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.988640    4555 cpu_manager.go:414] "containerMap: removing container" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main"
Oct 31 00:04:10.988691 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.988646    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main"
Oct 31 00:04:10.988894 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.988698    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1f96f9-25ce-489e-8f6d-4788685c83a0" containerName="revoke-keys"
Oct 31 00:04:10.988894 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.988710    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="3943f43f-9105-48e1-b60b-2ca22b65aa80" containerName="istio-proxy"
Oct 31 00:04:10.988894 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.988716    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="36867ec7-b651-4774-b248-c3f6b6696c69" containerName="main"
Oct 31 00:04:10.993591 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:10.993570    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.022232 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.022194    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"]
Oct 31 00:04:11.041552 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.041524    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.041552 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.041555    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.041771 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.041581    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5fsk\" (UniqueName: \"kubernetes.io/projected/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-kube-api-access-w5fsk\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.041771 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.041648    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.041771 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.041710    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.041771 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.041747    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.041926 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.041892    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.041968 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.041938    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.042040 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.042023    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.052024 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.051999    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"]
Oct 31 00:04:11.055489 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.055472    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.103685 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.103653    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"]
Oct 31 00:04:11.143493 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143463    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.143493 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143502    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.143764 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143535    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.143764 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143565    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fsk\" (UniqueName: \"kubernetes.io/projected/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-kube-api-access-w5fsk\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.143764 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143594    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.143764 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143675    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.143764 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143715    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.143764 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143750    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ncxp\" (UniqueName: \"kubernetes.io/projected/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-kube-api-access-5ncxp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143775    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143810    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143862    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143898    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143925    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143946    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.143982    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.144009    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.144027    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.144062    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.144128 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.144112    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.145426 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.144157    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.145426 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.144581    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.145426 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.144679    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.145647 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.145602    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.147019 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.146998    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.147478 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.147441    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.173045 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.173004    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.173486 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.173465    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5fsk\" (UniqueName: \"kubernetes.io/projected/cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d-kube-api-access-w5fsk\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dlhf9\" (UID: \"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.245117 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245044    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245117 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245092    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245322 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245148    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ncxp\" (UniqueName: \"kubernetes.io/projected/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-kube-api-access-5ncxp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245322 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245174    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245322 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245303    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245547 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245388    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245547 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245459    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245547 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245484    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245547 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245534    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245786 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245486    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245786 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245655    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245905 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245782    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245905 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245847    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.245999 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.245910    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.247799 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.247781    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.248110 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.248089    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.257175 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.257145    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.257406 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.257391    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ncxp\" (UniqueName: \"kubernetes.io/projected/330a67ac-e5a9-4bb7-9d70-89bfd52a5efe-kube-api-access-5ncxp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dghms\" (UID: \"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.305987 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.305960    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:11.368776 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.368741    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:11.589664 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.589636    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"]
Oct 31 00:04:11.590092 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:04:11.590071    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc70aa92_8b9e_4cf7_a66c_23b16fbcfd5d.slice/crio-e9a111cafcb1fb17514b7bb308531e913e68db50c3317d5478f71df58e280ba3 WatchSource:0}: Error finding container e9a111cafcb1fb17514b7bb308531e913e68db50c3317d5478f71df58e280ba3: Status 404 returned error can't find the container with id e9a111cafcb1fb17514b7bb308531e913e68db50c3317d5478f71df58e280ba3
Oct 31 00:04:11.592291 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.592262    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:04:11.592386 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.592325    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:04:11.592386 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.592356    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:04:11.653999 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.653970    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"]
Oct 31 00:04:11.654132 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:04:11.654106    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330a67ac_e5a9_4bb7_9d70_89bfd52a5efe.slice/crio-fc01c7f53f7b066539495f0078416f37c3b10b8b417202facc4f62b7ff9a5545 WatchSource:0}: Error finding container fc01c7f53f7b066539495f0078416f37c3b10b8b417202facc4f62b7ff9a5545: Status 404 returned error can't find the container with id fc01c7f53f7b066539495f0078416f37c3b10b8b417202facc4f62b7ff9a5545
Oct 31 00:04:11.656211 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.656179    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:04:11.656329 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.656241    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:04:11.656329 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:11.656270    4555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:04:12.107424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.106782    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" event={"ID":"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe","Type":"ContainerStarted","Data":"9e445d6c60cda482181d52fc43e795f1cfdf8f8b5177e37b4830092f5a941ed8"}
Oct 31 00:04:12.107424 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.106860    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" event={"ID":"330a67ac-e5a9-4bb7-9d70-89bfd52a5efe","Type":"ContainerStarted","Data":"fc01c7f53f7b066539495f0078416f37c3b10b8b417202facc4f62b7ff9a5545"}
Oct 31 00:04:12.111049 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.110929    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" event={"ID":"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d","Type":"ContainerStarted","Data":"b4cf65aef686f9ed16ba53bde4c445612b27f44849e76b841890a51eebca1a23"}
Oct 31 00:04:12.111049 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.111007    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" event={"ID":"cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d","Type":"ContainerStarted","Data":"e9a111cafcb1fb17514b7bb308531e913e68db50c3317d5478f71df58e280ba3"}
Oct 31 00:04:12.134756 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.134707    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" podStartSLOduration=2.134694715 podStartE2EDuration="2.134694715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:12.133561974 +0000 UTC m=+2711.060199555" watchObservedRunningTime="2026-10-31 00:04:12.134694715 +0000 UTC m=+2711.061332275"
Oct 31 00:04:12.157342 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.157296    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" podStartSLOduration=2.1572839520000002 podStartE2EDuration="2.157283952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:12.155573561 +0000 UTC m=+2711.082211146" watchObservedRunningTime="2026-10-31 00:04:12.157283952 +0000 UTC m=+2711.083921533"
Oct 31 00:04:12.307864 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.307782    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:12.311396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.309825    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dlhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 00:04:12.311396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.309885    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" podUID="cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 00:04:12.371141 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.369533    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:12.375533 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.375498    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dghms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 00:04:12.375830 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:12.375792    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" podUID="330a67ac-e5a9-4bb7-9d70-89bfd52a5efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 00:04:13.311401 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:13.310855    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dlhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 00:04:13.311401 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:13.310919    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" podUID="cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 00:04:13.370823 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:13.370782    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dghms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 00:04:13.371106 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:13.371075    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" podUID="330a67ac-e5a9-4bb7-9d70-89bfd52a5efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 00:04:14.310396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:14.309831    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dlhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 00:04:14.310396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:14.309897    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" podUID="cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 00:04:14.369854 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:14.369815    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dghms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 00:04:14.370454 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:14.370423    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" podUID="330a67ac-e5a9-4bb7-9d70-89bfd52a5efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 00:04:15.307396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:15.306840    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dlhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 00:04:15.307396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:15.306909    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" podUID="cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 00:04:15.370397 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:15.369786    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dghms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 00:04:15.370397 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:15.369850    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" podUID="330a67ac-e5a9-4bb7-9d70-89bfd52a5efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 00:04:16.311415 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:16.309785    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dlhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 00:04:16.311415 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:16.309846    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" podUID="cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 00:04:16.369794 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:16.369757    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dghms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 00:04:16.370090 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:16.370065    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" podUID="330a67ac-e5a9-4bb7-9d70-89bfd52a5efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 00:04:17.306858 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:17.306818    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dlhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 00:04:17.307503 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:17.307469    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" podUID="cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 00:04:17.369844 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:17.369807    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dghms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 00:04:17.370093 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:17.370066    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" podUID="330a67ac-e5a9-4bb7-9d70-89bfd52a5efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 00:04:18.307253 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:18.307212    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dlhf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 00:04:18.307677 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:18.307269    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" podUID="cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 00:04:18.369229 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:18.369196    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dghms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 00:04:18.369402 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:18.369250    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" podUID="330a67ac-e5a9-4bb7-9d70-89bfd52a5efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 00:04:19.369218 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:19.369168    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dghms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 00:04:19.369826 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:19.369236    4555 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms" podUID="330a67ac-e5a9-4bb7-9d70-89bfd52a5efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 00:04:19.962716 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:19.962684    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:19.963183 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:19.963159    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:19.964125 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:19.964097    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9"
Oct 31 00:04:20.751132 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:20.751094    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:20.751631 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:20.751540    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:20.752663 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:20.752639    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dghms"
Oct 31 00:04:27.447156 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:04:27.447111    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:04:32.026140 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:32.026105    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"]
Oct 31 00:04:32.030854 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:32.030825    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25af825-mmdfx"]
Oct 31 00:04:33.206039 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:33.206005    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f1254c-9f27-4f46-88f1-99650f88a59b" path="/var/lib/kubelet/pods/a9f1254c-9f27-4f46-88f1-99650f88a59b/volumes"
Oct 31 00:04:41.023437 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:41.023395    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"]
Oct 31 00:04:41.029275 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:41.029245    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b034523-4mbbl"]
Oct 31 00:04:41.206423 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:41.206390    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972e1861-195d-4c1b-b303-148291aa2c22" path="/var/lib/kubelet/pods/972e1861-195d-4c1b-b303-148291aa2c22/volumes"
Oct 31 00:04:50.147091 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:04:50.147054    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:04:57.446744 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:04:57.446706    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:05:04.261361 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:05:04.261322    4555 scope.go:122] "RemoveContainer" containerID="55521c29ccdaaf33e653b4a2f779f649f35e4bdde2f546e83cbca44e81cc2f2e"
Oct 31 00:05:04.270787 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:05:04.270762    4555 scope.go:122] "RemoveContainer" containerID="19a281b3c0ef6db3c536db71531dae407d71100f37874e86157d9ecd01b813f2"
Oct 31 00:05:04.279431 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:05:04.279408    4555 scope.go:122] "RemoveContainer" containerID="4fa928608c79a9243d4b24dd142f17ef7e3152aba1ed8cfc0884ae027c834f87"
Oct 31 00:05:04.287913 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:05:04.287892    4555 scope.go:122] "RemoveContainer" containerID="4e5e2082b32438d07b9f9bfb4f0930b81b4a25d4ea88345706393b070cc0dc70"
Oct 31 00:05:04.296412 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:05:04.296393    4555 scope.go:122] "RemoveContainer" containerID="bb43029c6b65eb341d308debe3fde347d349a2b40e70f334f750ce70a5c3a494"
Oct 31 00:05:04.304669 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:05:04.304646    4555 scope.go:122] "RemoveContainer" containerID="f3bf028131c1b0763ea6a4f660e2305c58aa546d18f09afa75f0be3b60fe5c6a"
Oct 31 00:05:05.827796 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:05:05.827752    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:05:10.050234 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:05:10.050199    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:05:24.746228 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:05:24.746194    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:05:27.457255 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:05:27.457208    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:05:37.644148 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:05:37.644111    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:05:57.444964 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:05:57.444918    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:06:02.045075 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:02.045037    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:06:04.351178 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:04.351140    4555 scope.go:122] "RemoveContainer" containerID="85357229d19b2645e5cce7408525a6714f0f135e9e364e3a9206306d1cfb464f"
Oct 31 00:06:04.388398 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:04.386538    4555 scope.go:122] "RemoveContainer" containerID="7ff5981afb67083bc171c02fd2eca9943c4bacafcb12457437e93292fac5a09d"
Oct 31 00:06:04.448349 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:04.448314    4555 scope.go:122] "RemoveContainer" containerID="9ba57ef8046d95497ec2dbcc4e2c26b0b7d115507e53b939396fad9e8cb89979"
Oct 31 00:06:04.459985 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:04.459963    4555 scope.go:122] "RemoveContainer" containerID="dfa1c2b6c6a057e2c60426b6643732f10ea3a5134b16f3bb081c2cefa0c9ae1c"
Oct 31 00:06:04.472296 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:04.472275    4555 scope.go:122] "RemoveContainer" containerID="bb358653e89be232640b6bb291bdb6a6cef907df967f00f2d41ec5e53a4d6eb0"
Oct 31 00:06:04.485835 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:04.485558    4555 scope.go:122] "RemoveContainer" containerID="5d799773ad23b9672450ec2d98787821b498985004d1e5e3bbf0a90efc18539f"
Oct 31 00:06:12.347186 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:12.347109    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:06:16.560636 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:16.560596    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:06:27.445207 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:06:27.445172    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:06:33.271515 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:33.271474    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:06:36.996395 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:36.996236    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:06:37.031812 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:37.031205    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"]
Oct 31 00:06:37.054418 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:37.050546    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef74bfe-pjwmz"]
Oct 31 00:06:37.245742 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:37.245702    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1493b5fd-8940-438b-9063-81aa1ae5a1c3" path="/var/lib/kubelet/pods/1493b5fd-8940-438b-9063-81aa1ae5a1c3/volumes"
Oct 31 00:06:51.452338 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:51.452284    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:06:56.786527 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:06:56.786485    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:06:57.446041 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:06:57.445997    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:07:10.164451 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:10.164406    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:07:13.913395 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:13.913339    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:07:26.253871 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.253830    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"]
Oct 31 00:07:26.254296 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.254079    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9" podUID="4fe60fd8-3fa6-46ec-9447-f122274a818c" containerName="discovery" containerID="cri-o://ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa" gracePeriod=30
Oct 31 00:07:26.509569 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.509501    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 31 00:07:26.653713 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.653682    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-kubeconfig\") pod \"4fe60fd8-3fa6-46ec-9447-f122274a818c\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") "
Oct 31 00:07:26.653922 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.653724    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-ca-configmap\") pod \"4fe60fd8-3fa6-46ec-9447-f122274a818c\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") "
Oct 31 00:07:26.653922 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.653755    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-dns-cert\") pod \"4fe60fd8-3fa6-46ec-9447-f122274a818c\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") "
Oct 31 00:07:26.653922 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.653820    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-cacerts\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-cacerts\") pod \"4fe60fd8-3fa6-46ec-9447-f122274a818c\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") "
Oct 31 00:07:26.653922 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.653845    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe60fd8-3fa6-46ec-9447-f122274a818c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe60fd8-3fa6-46ec-9447-f122274a818c-local-certs\") pod \"4fe60fd8-3fa6-46ec-9447-f122274a818c\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") "
Oct 31 00:07:26.653922 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.653896    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-token\" (UniqueName: \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-token\") pod \"4fe60fd8-3fa6-46ec-9447-f122274a818c\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") "
Oct 31 00:07:26.654212 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.653943    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-kube-api-access-sqtbx\" (UniqueName: \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-kube-api-access-sqtbx\") pod \"4fe60fd8-3fa6-46ec-9447-f122274a818c\" (UID: \"4fe60fd8-3fa6-46ec-9447-f122274a818c\") "
Oct 31 00:07:26.654212 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.654159    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-ca-configmap" pod "4fe60fd8-3fa6-46ec-9447-f122274a818c" (UID: "4fe60fd8-3fa6-46ec-9447-f122274a818c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:07:26.654332 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.654299    4555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-ca-configmap\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.656478 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.656450    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-dns-cert" pod "4fe60fd8-3fa6-46ec-9447-f122274a818c" (UID: "4fe60fd8-3fa6-46ec-9447-f122274a818c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:07:26.656786 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.656754    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-kubeconfig" pod "4fe60fd8-3fa6-46ec-9447-f122274a818c" (UID: "4fe60fd8-3fa6-46ec-9447-f122274a818c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:07:26.656896 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.656676    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-cacerts" pod "4fe60fd8-3fa6-46ec-9447-f122274a818c" (UID: "4fe60fd8-3fa6-46ec-9447-f122274a818c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:07:26.656896 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.656748    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-token" pod "4fe60fd8-3fa6-46ec-9447-f122274a818c" (UID: "4fe60fd8-3fa6-46ec-9447-f122274a818c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:07:26.656980 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.656964    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-kube-api-access-sqtbx" pod "4fe60fd8-3fa6-46ec-9447-f122274a818c" (UID: "4fe60fd8-3fa6-46ec-9447-f122274a818c"). InnerVolumeSpecName "kube-api-access-sqtbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:07:26.657149 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.657122    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe60fd8-3fa6-46ec-9447-f122274a818c-local-certs" pod "4fe60fd8-3fa6-46ec-9447-f122274a818c" (UID: "4fe60fd8-3fa6-46ec-9447-f122274a818c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:26.755257 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.755226    4555 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-cacerts\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.755257 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.755252    4555 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe60fd8-3fa6-46ec-9447-f122274a818c-local-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.755257 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.755261    4555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-token\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.755539 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.755270    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqtbx\" (UniqueName: \"kubernetes.io/projected/4fe60fd8-3fa6-46ec-9447-f122274a818c-kube-api-access-sqtbx\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.755539 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.755280    4555 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-kubeconfig\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.755539 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:26.755289    4555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4fe60fd8-3fa6-46ec-9447-f122274a818c-istio-csr-dns-cert\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:27.371983 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:27.371936    4555 generic.go:356] "Generic (PLEG): container finished" podID="4fe60fd8-3fa6-46ec-9447-f122274a818c" containerID="ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa" exitCode=0
Oct 31 00:07:27.372520 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:27.372055    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"
Oct 31 00:07:27.372520 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:27.372087    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9" event={"ID":"4fe60fd8-3fa6-46ec-9447-f122274a818c","Type":"ContainerDied","Data":"ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa"}
Oct 31 00:07:27.372520 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:27.372148    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9" event={"ID":"4fe60fd8-3fa6-46ec-9447-f122274a818c","Type":"ContainerDied","Data":"ce9d00ee54ecdfa47aedc544944c4373884cfd74ef80d24883a37c4af8939702"}
Oct 31 00:07:27.372520 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:27.372163    4555 scope.go:122] "RemoveContainer" containerID="ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa"
Oct 31 00:07:27.386843 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:27.386820    4555 scope.go:122] "RemoveContainer" containerID="ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa"
Oct 31 00:07:27.387112 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:07:27.387092    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa\": container with ID starting with ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa not found: ID does not exist" containerID="ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa"
Oct 31 00:07:27.387186 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:27.387123    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa"} err="failed to get container status \"ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa\": rpc error: code = NotFound desc = could not find container \"ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa\": container with ID starting with ac4daeb1129120b044cf1da8023a5e47097342bcc23472dda03a920ee2657efa not found: ID does not exist"
Oct 31 00:07:27.391808 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:27.391781    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"]
Oct 31 00:07:27.397208 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:27.397182    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dnqr9"]
Oct 31 00:07:27.447791 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:07:27.447747    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:07:28.032431 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:28.032399    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:07:29.206159 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:29.206115    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe60fd8-3fa6-46ec-9447-f122274a818c" path="/var/lib/kubelet/pods/4fe60fd8-3fa6-46ec-9447-f122274a818c/volumes"
Oct 31 00:07:38.774219 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:38.774185    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:07:43.300923 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:07:43.295945    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:07:57.446312 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:07:57.446273    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:08:04.684862 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:08:04.684828    4555 scope.go:122] "RemoveContainer" containerID="d3a19f2b7975fbc3ec15c9ef675d34cb800470ce1ba0e85cb913f20f6ba042d1"
Oct 31 00:08:07.453759 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:08:07.453719    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:08:20.042915 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:08:20.042879    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"]
Oct 31 00:08:20.052835 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:08:20.052807    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9cfcd8-k65dt"]
Oct 31 00:08:21.206061 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:08:21.206025    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1f96f9-25ce-489e-8f6d-4788685c83a0" path="/var/lib/kubelet/pods/ae1f96f9-25ce-489e-8f6d-4788685c83a0/volumes"
Oct 31 00:08:25.765877 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:08:25.765838    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:08:27.470154 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:08:27.470115    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:08:36.030834 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:08:36.030797    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:08:52.672238 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:08:52.672202    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:08:57.446874 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:08:57.446819    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:08:57.460237 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:08:57.460208    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:09:10.671693 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:10.671653    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:09:18.247676 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.247591    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"]
Oct 31 00:09:18.248043 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.248007    4555 cpu_manager.go:414] "containerMap: removing container" podUID="4fe60fd8-3fa6-46ec-9447-f122274a818c" containerName="discovery"
Oct 31 00:09:18.248043 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.248021    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe60fd8-3fa6-46ec-9447-f122274a818c" containerName="discovery"
Oct 31 00:09:18.248121 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.248101    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe60fd8-3fa6-46ec-9447-f122274a818c" containerName="discovery"
Oct 31 00:09:18.251361 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.251340    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.264281 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.264253    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-93f1ad\"/\"openshift-service-ca.crt\""
Oct 31 00:09:18.264580 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.264551    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-93f1ad\"/\"kube-root-ca.crt\""
Oct 31 00:09:18.265554 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.265536    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-93f1ad\"/\"default-dockercfg-xxbmr\""
Oct 31 00:09:18.265687 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.265666    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-93f1ad\"/\"simulator-single-main-93f1ad-kserve-self-signed-certs\""
Oct 31 00:09:18.274180 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.274152    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"]
Oct 31 00:09:18.338428 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.338384    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-tmp-dir\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.338428 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.338429    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-model-cache\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.338668 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.338555    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed1dace-73ea-44b7-8677-5a888c640e11-tls-certs\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.338668 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.338653    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-home\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.338765 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.338715    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-522lb\" (UniqueName: \"kubernetes.io/projected/0ed1dace-73ea-44b7-8677-5a888c640e11-kube-api-access-522lb\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.338815 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.338767    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-dshm\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.439318 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.439277    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed1dace-73ea-44b7-8677-5a888c640e11-tls-certs\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.439557 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.439332    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-home\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.439557 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.439397    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-522lb\" (UniqueName: \"kubernetes.io/projected/0ed1dace-73ea-44b7-8677-5a888c640e11-kube-api-access-522lb\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.439557 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.439428    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-dshm\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.439557 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.439461    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-tmp-dir\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.439557 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.439486    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-model-cache\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.439873 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.439820    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-home\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.439934 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.439881    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-model-cache\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.439934 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.439917    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-tmp-dir\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.441990 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.441960    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-dshm\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.442103 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.442026    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed1dace-73ea-44b7-8677-5a888c640e11-tls-certs\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.509003 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.508910    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-522lb\" (UniqueName: \"kubernetes.io/projected/0ed1dace-73ea-44b7-8677-5a888c640e11-kube-api-access-522lb\") pod \"simulator-single-main-93f1ad-kserve-7799f677db-bp4q8\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") " pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.564700 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.564662    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:18.858167 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:18.858141    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"]
Oct 31 00:09:18.860017 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:09:18.859988    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ed1dace_73ea_44b7_8677_5a888c640e11.slice/crio-d7be58898d1c1f313dbdc75d0044842bad7e3d1e216f7159e85276f620d49b10 WatchSource:0}: Error finding container d7be58898d1c1f313dbdc75d0044842bad7e3d1e216f7159e85276f620d49b10: Status 404 returned error can't find the container with id d7be58898d1c1f313dbdc75d0044842bad7e3d1e216f7159e85276f620d49b10
Oct 31 00:09:19.031305 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:19.031269    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8" event={"ID":"0ed1dace-73ea-44b7-8677-5a888c640e11","Type":"ContainerStarted","Data":"826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672"}
Oct 31 00:09:19.031541 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:19.031320    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8" event={"ID":"0ed1dace-73ea-44b7-8677-5a888c640e11","Type":"ContainerStarted","Data":"d7be58898d1c1f313dbdc75d0044842bad7e3d1e216f7159e85276f620d49b10"}
Oct 31 00:09:19.051998 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:19.051940    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8" podStartSLOduration=1.051921827 podStartE2EDuration="1.051921827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:19.050861021 +0000 UTC m=+3017.977498645" watchObservedRunningTime="2026-10-31 00:09:19.051921827 +0000 UTC m=+3017.978559409"
Oct 31 00:09:19.565561 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:19.565526    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:20.160528 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:20.160489    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:09:27.446728 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:09:27.446689    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:09:28.565249 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:28.565215    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:28.577943 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:28.577915    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:29.094584 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:29.094554    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:09:36.331805 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:09:36.331765    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:09:57.444092 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:09:57.444053    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:10:00.342645 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:00.342610    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:10:04.523895 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:04.523859    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"]
Oct 31 00:10:04.718860 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:04.718819    4555 scope.go:122] "RemoveContainer" containerID="cb1ec9085f10bc8d5e49476e303db91001ac50f0feee54ab241101bcf7949501"
Oct 31 00:10:12.358411 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:12.358362    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:10:19.544024 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.543960    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8" podUID="0ed1dace-73ea-44b7-8677-5a888c640e11" containerName="main" containerID="cri-o://826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672" gracePeriod=45
Oct 31 00:10:19.798578 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.798512    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:10:19.945116 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.945081    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed1dace-73ea-44b7-8677-5a888c640e11-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed1dace-73ea-44b7-8677-5a888c640e11-tls-certs\") pod \"0ed1dace-73ea-44b7-8677-5a888c640e11\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") "
Oct 31 00:10:19.945299 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.945165    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-dshm\") pod \"0ed1dace-73ea-44b7-8677-5a888c640e11\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") "
Oct 31 00:10:19.945299 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.945211    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed1dace-73ea-44b7-8677-5a888c640e11-kube-api-access-522lb\" (UniqueName: \"kubernetes.io/projected/0ed1dace-73ea-44b7-8677-5a888c640e11-kube-api-access-522lb\") pod \"0ed1dace-73ea-44b7-8677-5a888c640e11\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") "
Oct 31 00:10:19.945299 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.945239    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-home\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-home\") pod \"0ed1dace-73ea-44b7-8677-5a888c640e11\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") "
Oct 31 00:10:19.945299 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.945294    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-model-cache\") pod \"0ed1dace-73ea-44b7-8677-5a888c640e11\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") "
Oct 31 00:10:19.945561 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.945381    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-tmp-dir\") pod \"0ed1dace-73ea-44b7-8677-5a888c640e11\" (UID: \"0ed1dace-73ea-44b7-8677-5a888c640e11\") "
Oct 31 00:10:19.945659 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.945626    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-home" pod "0ed1dace-73ea-44b7-8677-5a888c640e11" (UID: "0ed1dace-73ea-44b7-8677-5a888c640e11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:10:19.945716 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.945659    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-model-cache" pod "0ed1dace-73ea-44b7-8677-5a888c640e11" (UID: "0ed1dace-73ea-44b7-8677-5a888c640e11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:10:19.945754 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.945717    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-tmp-dir" pod "0ed1dace-73ea-44b7-8677-5a888c640e11" (UID: "0ed1dace-73ea-44b7-8677-5a888c640e11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:10:19.947630 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.947602    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed1dace-73ea-44b7-8677-5a888c640e11-kube-api-access-522lb" pod "0ed1dace-73ea-44b7-8677-5a888c640e11" (UID: "0ed1dace-73ea-44b7-8677-5a888c640e11"). InnerVolumeSpecName "kube-api-access-522lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:10:19.947838 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.947815    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-dshm" pod "0ed1dace-73ea-44b7-8677-5a888c640e11" (UID: "0ed1dace-73ea-44b7-8677-5a888c640e11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:10:19.947838 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:19.947826    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed1dace-73ea-44b7-8677-5a888c640e11-tls-certs" pod "0ed1dace-73ea-44b7-8677-5a888c640e11" (UID: "0ed1dace-73ea-44b7-8677-5a888c640e11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:10:20.046885 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.046844    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:20.046885 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.046876    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed1dace-73ea-44b7-8677-5a888c640e11-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:20.046885 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.046884    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:20.046885 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.046893    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-522lb\" (UniqueName: \"kubernetes.io/projected/0ed1dace-73ea-44b7-8677-5a888c640e11-kube-api-access-522lb\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:20.047170 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.046903    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:20.047170 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.046911    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed1dace-73ea-44b7-8677-5a888c640e11-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:20.363492 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.363452    4555 generic.go:356] "Generic (PLEG): container finished" podID="0ed1dace-73ea-44b7-8677-5a888c640e11" containerID="826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672" exitCode=0
Oct 31 00:10:20.363680 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.363507    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"
Oct 31 00:10:20.363680 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.363536    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8" event={"ID":"0ed1dace-73ea-44b7-8677-5a888c640e11","Type":"ContainerDied","Data":"826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672"}
Oct 31 00:10:20.363680 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.363582    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8" event={"ID":"0ed1dace-73ea-44b7-8677-5a888c640e11","Type":"ContainerDied","Data":"d7be58898d1c1f313dbdc75d0044842bad7e3d1e216f7159e85276f620d49b10"}
Oct 31 00:10:20.363680 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.363599    4555 scope.go:122] "RemoveContainer" containerID="826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672"
Oct 31 00:10:20.373314 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.373295    4555 scope.go:122] "RemoveContainer" containerID="826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672"
Oct 31 00:10:20.373591 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:10:20.373571    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672\": container with ID starting with 826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672 not found: ID does not exist" containerID="826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672"
Oct 31 00:10:20.373648 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.373600    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672"} err="failed to get container status \"826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672\": rpc error: code = NotFound desc = could not find container \"826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672\": container with ID starting with 826b071cfe6843c65e259ee5271a169026114f254f6b525a8c22999009576672 not found: ID does not exist"
Oct 31 00:10:20.386346 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.386322    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"]
Oct 31 00:10:20.392653 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:20.392632    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-93f1ad/simulator-single-main-93f1ad-kserve-7799f677db-bp4q8"]
Oct 31 00:10:21.206205 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:21.206172    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed1dace-73ea-44b7-8677-5a888c640e11" path="/var/lib/kubelet/pods/0ed1dace-73ea-44b7-8677-5a888c640e11/volumes"
Oct 31 00:10:27.445325 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:10:27.445289    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:10:34.993635 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:34.993593    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"]
Oct 31 00:10:34.994194 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:34.993834    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7" podUID="8b3411a0-327b-4919-8084-a07db4a5c4e8" containerName="manager" containerID="cri-o://6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d" gracePeriod=10
Oct 31 00:10:35.241954 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.241928    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 31 00:10:35.280565 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.280539    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3411a0-327b-4919-8084-a07db4a5c4e8-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3411a0-327b-4919-8084-a07db4a5c4e8-extensions-socket-volume\") pod \"8b3411a0-327b-4919-8084-a07db4a5c4e8\" (UID: \"8b3411a0-327b-4919-8084-a07db4a5c4e8\") "
Oct 31 00:10:35.280727 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.280667    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3411a0-327b-4919-8084-a07db4a5c4e8-kube-api-access-slw8p\" (UniqueName: \"kubernetes.io/projected/8b3411a0-327b-4919-8084-a07db4a5c4e8-kube-api-access-slw8p\") pod \"8b3411a0-327b-4919-8084-a07db4a5c4e8\" (UID: \"8b3411a0-327b-4919-8084-a07db4a5c4e8\") "
Oct 31 00:10:35.281033 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.281005    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3411a0-327b-4919-8084-a07db4a5c4e8-extensions-socket-volume" pod "8b3411a0-327b-4919-8084-a07db4a5c4e8" (UID: "8b3411a0-327b-4919-8084-a07db4a5c4e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:10:35.282992 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.282953    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3411a0-327b-4919-8084-a07db4a5c4e8-kube-api-access-slw8p" pod "8b3411a0-327b-4919-8084-a07db4a5c4e8" (UID: "8b3411a0-327b-4919-8084-a07db4a5c4e8"). InnerVolumeSpecName "kube-api-access-slw8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:10:35.382064 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.382028    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slw8p\" (UniqueName: \"kubernetes.io/projected/8b3411a0-327b-4919-8084-a07db4a5c4e8-kube-api-access-slw8p\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:35.382064 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.382057    4555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3411a0-327b-4919-8084-a07db4a5c4e8-extensions-socket-volume\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:10:35.446730 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.446693    4555 generic.go:356] "Generic (PLEG): container finished" podID="8b3411a0-327b-4919-8084-a07db4a5c4e8" containerID="6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d" exitCode=0
Oct 31 00:10:35.446919 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.446744    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"
Oct 31 00:10:35.446919 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.446767    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7" event={"ID":"8b3411a0-327b-4919-8084-a07db4a5c4e8","Type":"ContainerDied","Data":"6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d"}
Oct 31 00:10:35.446919 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.446824    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7" event={"ID":"8b3411a0-327b-4919-8084-a07db4a5c4e8","Type":"ContainerDied","Data":"ef7360e4f94e002764c5db76de9f62292c15b3b1a58e0f182af5b699b0a951a0"}
Oct 31 00:10:35.446919 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.446851    4555 scope.go:122] "RemoveContainer" containerID="6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d"
Oct 31 00:10:35.458258 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.458239    4555 scope.go:122] "RemoveContainer" containerID="6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d"
Oct 31 00:10:35.458575 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:10:35.458555    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d\": container with ID starting with 6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d not found: ID does not exist" containerID="6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d"
Oct 31 00:10:35.458644 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.458583    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d"} err="failed to get container status \"6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d\": rpc error: code = NotFound desc = could not find container \"6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d\": container with ID starting with 6057e0433d06703dcde4cb698435765928924b5188610e235aaba550afc46c7d not found: ID does not exist"
Oct 31 00:10:35.471220 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.471191    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"]
Oct 31 00:10:35.473003 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:35.472980    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwfw7"]
Oct 31 00:10:37.206023 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:10:37.205985    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3411a0-327b-4919-8084-a07db4a5c4e8" path="/var/lib/kubelet/pods/8b3411a0-327b-4919-8084-a07db4a5c4e8/volumes"
Oct 31 00:10:57.446251 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:10:57.446212    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:11:27.445744 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:11:27.445707    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:11:36.085331 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.085297    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"]
Oct 31 00:11:36.085961 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.085842    4555 cpu_manager.go:414] "containerMap: removing container" podUID="8b3411a0-327b-4919-8084-a07db4a5c4e8" containerName="manager"
Oct 31 00:11:36.085961 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.085862    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3411a0-327b-4919-8084-a07db4a5c4e8" containerName="manager"
Oct 31 00:11:36.085961 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.085916    4555 cpu_manager.go:414] "containerMap: removing container" podUID="0ed1dace-73ea-44b7-8677-5a888c640e11" containerName="main"
Oct 31 00:11:36.085961 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.085924    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed1dace-73ea-44b7-8677-5a888c640e11" containerName="main"
Oct 31 00:11:36.086197 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.085999    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3411a0-327b-4919-8084-a07db4a5c4e8" containerName="manager"
Oct 31 00:11:36.086197 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.086019    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed1dace-73ea-44b7-8677-5a888c640e11" containerName="main"
Oct 31 00:11:36.089262 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.089241    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"
Oct 31 00:11:36.101686 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.101663    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4vmw2\""
Oct 31 00:11:36.102675 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.102651    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"]
Oct 31 00:11:36.122595 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.122567    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkbhr\" (UniqueName: \"kubernetes.io/projected/3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54-kube-api-access-wkbhr\") pod \"kuadrant-operator-controller-manager-678d87766f-tzfff\" (UID: \"3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"
Oct 31 00:11:36.122744 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.122602    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tzfff\" (UID: \"3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"
Oct 31 00:11:36.223455 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.223419    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkbhr\" (UniqueName: \"kubernetes.io/projected/3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54-kube-api-access-wkbhr\") pod \"kuadrant-operator-controller-manager-678d87766f-tzfff\" (UID: \"3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"
Oct 31 00:11:36.223455 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.223460    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tzfff\" (UID: \"3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"
Oct 31 00:11:36.223893 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.223873    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tzfff\" (UID: \"3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"
Oct 31 00:11:36.236420 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.236356    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkbhr\" (UniqueName: \"kubernetes.io/projected/3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54-kube-api-access-wkbhr\") pod \"kuadrant-operator-controller-manager-678d87766f-tzfff\" (UID: \"3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"
Oct 31 00:11:36.404116 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.404029    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"
Oct 31 00:11:36.690756 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.690630    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"]
Oct 31 00:11:36.693256 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:11:36.693228    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb30b4a_dc12_4cbc_be11_bfcf8ab79b54.slice/crio-9b855c601e2a1062e15fc68ac42dcdad9f4599faa6acac44cdc48a6efade179d WatchSource:0}: Error finding container 9b855c601e2a1062e15fc68ac42dcdad9f4599faa6acac44cdc48a6efade179d: Status 404 returned error can't find the container with id 9b855c601e2a1062e15fc68ac42dcdad9f4599faa6acac44cdc48a6efade179d
Oct 31 00:11:36.753853 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:36.753814    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff" event={"ID":"3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54","Type":"ContainerStarted","Data":"9b855c601e2a1062e15fc68ac42dcdad9f4599faa6acac44cdc48a6efade179d"}
Oct 31 00:11:37.760285 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:37.760236    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff" event={"ID":"3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54","Type":"ContainerStarted","Data":"b6a3208e83cc0a9fc06709b50104dd1a9fd2e61b3f8d40d01a37a97bf2285723"}
Oct 31 00:11:37.760715 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:37.760329    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"
Oct 31 00:11:37.794979 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:37.794914    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff" podStartSLOduration=1.794898095 podStartE2EDuration="1.794898095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:37.793305175 +0000 UTC m=+3156.719942756" watchObservedRunningTime="2026-10-31 00:11:37.794898095 +0000 UTC m=+3156.721535654"
Oct 31 00:11:48.767868 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:48.767824    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tzfff"
Oct 31 00:11:54.153970 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:11:54.153935    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:11:57.446411 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:11:57.446345    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:12:09.131753 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:09.131715    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:12:13.946218 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:13.946052    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:12:27.445842 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:12:27.445803    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:12:28.159642 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:28.159606    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:12:36.386017 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:36.385978    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:12:50.964199 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:50.964118    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"]
Oct 31 00:12:50.968341 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:50.968319    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:50.976539 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:50.976513    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c4c1ab\"/\"kube-root-ca.crt\""
Oct 31 00:12:50.976539 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:50.976513    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c4c1ab\"/\"simulator-single-main-c4c1ab-kserve-self-signed-certs\""
Oct 31 00:12:50.978462 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:50.977336    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c4c1ab\"/\"default-dockercfg-mst7n\""
Oct 31 00:12:50.978462 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:50.977403    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c4c1ab\"/\"openshift-service-ca.crt\""
Oct 31 00:12:50.980153 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:50.980102    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"]
Oct 31 00:12:51.065531 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.065485    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-dshm\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.065531 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.065534    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-model-cache\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.065826 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.065640    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81c5894-c8f7-4b30-928a-2a300c65470c-tls-certs\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.065826 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.065720    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxsjb\" (UniqueName: \"kubernetes.io/projected/c81c5894-c8f7-4b30-928a-2a300c65470c-kube-api-access-pxsjb\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.065826 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.065770    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-tmp-dir\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.065981 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.065842    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-home\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.166646 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.166585    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-dshm\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.166829 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.166682    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-model-cache\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.166829 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.166765    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81c5894-c8f7-4b30-928a-2a300c65470c-tls-certs\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.166829 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.166822    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxsjb\" (UniqueName: \"kubernetes.io/projected/c81c5894-c8f7-4b30-928a-2a300c65470c-kube-api-access-pxsjb\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.166957 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.166854    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-tmp-dir\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.166957 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.166889    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-home\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.167109 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.167088    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-model-cache\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.167288 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.167263    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-home\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.167530 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.167505    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-tmp-dir\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.169653 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.169628    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-dshm\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.170720 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.170695    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81c5894-c8f7-4b30-928a-2a300c65470c-tls-certs\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.179302 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.179276    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxsjb\" (UniqueName: \"kubernetes.io/projected/c81c5894-c8f7-4b30-928a-2a300c65470c-kube-api-access-pxsjb\") pod \"simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") " pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.284010 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.283971    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:51.568435 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:51.568397    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"]
Oct 31 00:12:51.570852 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:12:51.570826    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc81c5894_c8f7_4b30_928a_2a300c65470c.slice/crio-5ccffd8a27a12be91855d6051d50ce2809b74da7ada4db9dfb3b69b5d71f9750 WatchSource:0}: Error finding container 5ccffd8a27a12be91855d6051d50ce2809b74da7ada4db9dfb3b69b5d71f9750: Status 404 returned error can't find the container with id 5ccffd8a27a12be91855d6051d50ce2809b74da7ada4db9dfb3b69b5d71f9750
Oct 31 00:12:52.161834 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:52.161793    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv" event={"ID":"c81c5894-c8f7-4b30-928a-2a300c65470c","Type":"ContainerStarted","Data":"ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518"}
Oct 31 00:12:52.162289 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:52.161850    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv" event={"ID":"c81c5894-c8f7-4b30-928a-2a300c65470c","Type":"ContainerStarted","Data":"5ccffd8a27a12be91855d6051d50ce2809b74da7ada4db9dfb3b69b5d71f9750"}
Oct 31 00:12:52.182030 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:52.181981    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv" podStartSLOduration=2.18196454 podStartE2EDuration="2.18196454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:52.180947741 +0000 UTC m=+3231.107585323" watchObservedRunningTime="2026-10-31 00:12:52.18196454 +0000 UTC m=+3231.108602118"
Oct 31 00:12:52.284324 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:52.284293    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:12:52.790490 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:12:52.790455    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:12:57.445784 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:12:57.445742    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:13:01.284187 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:01.284159    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:13:01.297031 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:01.297008    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:13:02.228882 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:02.228849    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:13:08.765947 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:08.765904    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:13:27.445547 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:13:27.445507    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:13:31.639939 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:31.639903    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:13:36.438719 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:36.438677    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"]
Oct 31 00:13:43.851085 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:43.850989    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:13:51.459523 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.459479    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv" podUID="c81c5894-c8f7-4b30-928a-2a300c65470c" containerName="main" containerID="cri-o://ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518" gracePeriod=45
Oct 31 00:13:51.706554 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.706526    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:13:51.731675 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.731596    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-model-cache\") pod \"c81c5894-c8f7-4b30-928a-2a300c65470c\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") "
Oct 31 00:13:51.731675 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.731647    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-dshm\") pod \"c81c5894-c8f7-4b30-928a-2a300c65470c\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") "
Oct 31 00:13:51.731921 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.731679    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81c5894-c8f7-4b30-928a-2a300c65470c-tls-certs\" (UniqueName: \"kubernetes.io/secret/c81c5894-c8f7-4b30-928a-2a300c65470c-tls-certs\") pod \"c81c5894-c8f7-4b30-928a-2a300c65470c\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") "
Oct 31 00:13:51.731921 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.731713    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81c5894-c8f7-4b30-928a-2a300c65470c-kube-api-access-pxsjb\" (UniqueName: \"kubernetes.io/projected/c81c5894-c8f7-4b30-928a-2a300c65470c-kube-api-access-pxsjb\") pod \"c81c5894-c8f7-4b30-928a-2a300c65470c\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") "
Oct 31 00:13:51.731921 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.731737    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-tmp-dir\") pod \"c81c5894-c8f7-4b30-928a-2a300c65470c\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") "
Oct 31 00:13:51.732108 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.731918    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-model-cache" pod "c81c5894-c8f7-4b30-928a-2a300c65470c" (UID: "c81c5894-c8f7-4b30-928a-2a300c65470c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:51.732108 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.731996    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:51.732229 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.732114    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-tmp-dir" pod "c81c5894-c8f7-4b30-928a-2a300c65470c" (UID: "c81c5894-c8f7-4b30-928a-2a300c65470c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:51.734893 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.734865    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81c5894-c8f7-4b30-928a-2a300c65470c-tls-certs" pod "c81c5894-c8f7-4b30-928a-2a300c65470c" (UID: "c81c5894-c8f7-4b30-928a-2a300c65470c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:13:51.735034 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.734935    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-dshm" pod "c81c5894-c8f7-4b30-928a-2a300c65470c" (UID: "c81c5894-c8f7-4b30-928a-2a300c65470c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:51.735985 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.735959    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81c5894-c8f7-4b30-928a-2a300c65470c-kube-api-access-pxsjb" pod "c81c5894-c8f7-4b30-928a-2a300c65470c" (UID: "c81c5894-c8f7-4b30-928a-2a300c65470c"). InnerVolumeSpecName "kube-api-access-pxsjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:13:51.832861 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.832830    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-home\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-home\") pod \"c81c5894-c8f7-4b30-928a-2a300c65470c\" (UID: \"c81c5894-c8f7-4b30-928a-2a300c65470c\") "
Oct 31 00:13:51.833065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.833012    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81c5894-c8f7-4b30-928a-2a300c65470c-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:51.833065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.833025    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxsjb\" (UniqueName: \"kubernetes.io/projected/c81c5894-c8f7-4b30-928a-2a300c65470c-kube-api-access-pxsjb\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:51.833065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.833035    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:51.833065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.833043    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:51.833263 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.833133    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-home" pod "c81c5894-c8f7-4b30-928a-2a300c65470c" (UID: "c81c5894-c8f7-4b30-928a-2a300c65470c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:51.934085 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:51.934036    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81c5894-c8f7-4b30-928a-2a300c65470c-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:52.477238 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:52.477197    4555 generic.go:356] "Generic (PLEG): container finished" podID="c81c5894-c8f7-4b30-928a-2a300c65470c" containerID="ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518" exitCode=0
Oct 31 00:13:52.477701 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:52.477246    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"
Oct 31 00:13:52.477701 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:52.477285    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv" event={"ID":"c81c5894-c8f7-4b30-928a-2a300c65470c","Type":"ContainerDied","Data":"ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518"}
Oct 31 00:13:52.477701 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:52.477337    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv" event={"ID":"c81c5894-c8f7-4b30-928a-2a300c65470c","Type":"ContainerDied","Data":"5ccffd8a27a12be91855d6051d50ce2809b74da7ada4db9dfb3b69b5d71f9750"}
Oct 31 00:13:52.477701 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:52.477355    4555 scope.go:122] "RemoveContainer" containerID="ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518"
Oct 31 00:13:52.489414 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:52.489341    4555 scope.go:122] "RemoveContainer" containerID="ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518"
Oct 31 00:13:52.489766 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:13:52.489742    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518\": container with ID starting with ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518 not found: ID does not exist" containerID="ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518"
Oct 31 00:13:52.489869 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:52.489776    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518"} err="failed to get container status \"ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518\": rpc error: code = NotFound desc = could not find container \"ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518\": container with ID starting with ea9cc9785a8f8145c86850f9a1ac1d4b31b7893645b2b118935dace551cc4518 not found: ID does not exist"
Oct 31 00:13:52.499937 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:52.499914    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"]
Oct 31 00:13:52.505820 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:52.505799    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c4c1ab/simulator-single-main-c4c1ab-kserve-67db45dc6b-vpltv"]
Oct 31 00:13:53.206470 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.206433    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81c5894-c8f7-4b30-928a-2a300c65470c" path="/var/lib/kubelet/pods/c81c5894-c8f7-4b30-928a-2a300c65470c/volumes"
Oct 31 00:13:53.790903 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.790871    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"]
Oct 31 00:13:53.791307 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.791254    4555 cpu_manager.go:414] "containerMap: removing container" podUID="c81c5894-c8f7-4b30-928a-2a300c65470c" containerName="main"
Oct 31 00:13:53.791307 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.791292    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81c5894-c8f7-4b30-928a-2a300c65470c" containerName="main"
Oct 31 00:13:53.791528 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.791507    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81c5894-c8f7-4b30-928a-2a300c65470c" containerName="main"
Oct 31 00:13:53.796566 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.796548    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.803337 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.803314    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v5mxr\""
Oct 31 00:13:53.804461 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.804439    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"]
Oct 31 00:13:53.814943 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.814920    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 00:13:53.814943 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.814948    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 00:13:53.850951 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.850919    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxc8m\" (UniqueName: \"kubernetes.io/projected/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-kube-api-access-kxc8m\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.851164 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.851009    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-dshm\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.851164 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.851031    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-home\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.851164 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.851055    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tls-certs\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.851164 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.851079    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-model-cache\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.851431 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.851198    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tmp-dir\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.902859 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.902826    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a30f2db8-kserve-self-signed-certs\""
Oct 31 00:13:53.954687 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.954649    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-dshm\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.954984 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.954694    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-home\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.954984 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.954737    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tls-certs\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.954984 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.954766    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-model-cache\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.954984 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.954831    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tmp-dir\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.954984 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.954899    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxc8m\" (UniqueName: \"kubernetes.io/projected/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-kube-api-access-kxc8m\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.955768 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.955742    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tmp-dir\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.955987 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.955875    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-model-cache\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.956138 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.956104    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-home\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.958454 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.958434    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-dshm\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.958686 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.958666    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tls-certs\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:53.969112 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:53.969089    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxc8m\" (UniqueName: \"kubernetes.io/projected/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-kube-api-access-kxc8m\") pod \"e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") " pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:54.113101 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:54.113007    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:54.395937 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:54.395913    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"]
Oct 31 00:13:54.398306 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:13:54.398275    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f52a91f_7d8b_4e5d_8cf4_f27821c8835e.slice/crio-6b64324d163022f85d093e9516715d7fb29afdff958c5547dcf302759f29ea45 WatchSource:0}: Error finding container 6b64324d163022f85d093e9516715d7fb29afdff958c5547dcf302759f29ea45: Status 404 returned error can't find the container with id 6b64324d163022f85d093e9516715d7fb29afdff958c5547dcf302759f29ea45
Oct 31 00:13:54.491765 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:54.491724    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr" event={"ID":"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e","Type":"ContainerStarted","Data":"227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f"}
Oct 31 00:13:54.491953 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:54.491777    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr" event={"ID":"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e","Type":"ContainerStarted","Data":"6b64324d163022f85d093e9516715d7fb29afdff958c5547dcf302759f29ea45"}
Oct 31 00:13:54.512261 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:54.512212    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr" podStartSLOduration=1.512198234 podStartE2EDuration="1.512198234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:54.510976056 +0000 UTC m=+3293.437613638" watchObservedRunningTime="2026-10-31 00:13:54.512198234 +0000 UTC m=+3293.438835812"
Oct 31 00:13:55.114032 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:55.114000    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:13:55.536901 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:13:55.536863    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:13:57.513690 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:13:57.513647    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:14:04.113790 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:04.113752    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:14:04.126241 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:04.126217    4555 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:14:04.575141 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:04.575112    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:14:11.966270 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:11.966223    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:14:27.445470 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:14:27.445427    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:14:27.978737 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:27.978690    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:14:31.233391 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:31.232472    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-mjk4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:14:31.233391 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:31.232538    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-mjk4b" podUID="ea819a48-154f-4c7a-a747-3306ac423289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:14:38.256663 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:38.256622    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"]
Oct 31 00:14:39.882101 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:39.882066    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:14:45.236827 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:45.236793    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:14:51.928716 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:51.928681    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:14:53.289000 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.288954    4555 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr" podUID="4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" containerName="main" containerID="cri-o://227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f" gracePeriod=45
Oct 31 00:14:53.793108 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.792578    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:14:53.837767 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.837664    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-dshm\") pod \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") "
Oct 31 00:14:53.837767 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.837740    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-model-cache\") pod \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") "
Oct 31 00:14:53.838037 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.837783    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-home\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-home\") pod \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") "
Oct 31 00:14:53.838037 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.837858    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tmp-dir\") pod \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") "
Oct 31 00:14:53.838037 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.837954    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-kube-api-access-kxc8m\" (UniqueName: \"kubernetes.io/projected/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-kube-api-access-kxc8m\") pod \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") "
Oct 31 00:14:53.838037 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.837986    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tls-certs\") pod \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\" (UID: \"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e\") "
Oct 31 00:14:53.839016 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.838988    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-home" pod "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" (UID: "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:53.839880 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.839646    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tmp-dir" pod "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" (UID: "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:53.839880 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.839717    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-model-cache" pod "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" (UID: "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:53.850927 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.850718    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tls-certs" pod "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" (UID: "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:14:53.850927 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.850874    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-dshm" pod "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" (UID: "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:53.851415 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.851341    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-kube-api-access-kxc8m" pod "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" (UID: "4f52a91f-7d8b-4e5d-8cf4-f27821c8835e"). InnerVolumeSpecName "kube-api-access-kxc8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:53.938990 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.938930    4555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-home\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:53.938990 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.938971    4555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tmp-dir\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:53.938990 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.938987    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxc8m\" (UniqueName: \"kubernetes.io/projected/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-kube-api-access-kxc8m\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:53.939360 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.939005    4555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-tls-certs\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:53.939360 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.939019    4555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-dshm\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:53.939360 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.939034    4555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e-model-cache\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:53.939629 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.939558    4555 generic.go:356] "Generic (PLEG): container finished" podID="4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" containerID="227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f" exitCode=0
Oct 31 00:14:53.940400 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.939711    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr" event={"ID":"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e","Type":"ContainerDied","Data":"227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f"}
Oct 31 00:14:53.940400 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.939759    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr" event={"ID":"4f52a91f-7d8b-4e5d-8cf4-f27821c8835e","Type":"ContainerDied","Data":"6b64324d163022f85d093e9516715d7fb29afdff958c5547dcf302759f29ea45"}
Oct 31 00:14:53.940400 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.939785    4555 scope.go:122] "RemoveContainer" containerID="227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f"
Oct 31 00:14:53.940400 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.939956    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"
Oct 31 00:14:53.959095 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.959075    4555 scope.go:122] "RemoveContainer" containerID="227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f"
Oct 31 00:14:53.959477 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:14:53.959451    4555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f\": container with ID starting with 227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f not found: ID does not exist" containerID="227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f"
Oct 31 00:14:53.959552 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.959490    4555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f"} err="failed to get container status \"227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f\": rpc error: code = NotFound desc = could not find container \"227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f\": container with ID starting with 227305e284baf3268755c82d482fe5b7ff062e2e2e8794d4791b46d20d1dc02f not found: ID does not exist"
Oct 31 00:14:53.977278 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.977231    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"]
Oct 31 00:14:53.982236 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:53.982175    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr"]
Oct 31 00:14:54.066397 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:14:54.064058    4555 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f52a91f_7d8b_4e5d_8cf4_f27821c8835e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f52a91f_7d8b_4e5d_8cf4_f27821c8835e.slice/crio-6b64324d163022f85d093e9516715d7fb29afdff958c5547dcf302759f29ea45\": RecentStats: unable to find data in memory cache]"
Oct 31 00:14:54.608563 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:54.608458    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-a30f2db8-kserve-9d79fb4bf-84pkr" podUID="4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:14:55.209091 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:55.209040    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" path="/var/lib/kubelet/pods/4f52a91f-7d8b-4e5d-8cf4-f27821c8835e/volumes"
Oct 31 00:14:57.171057 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:14:57.171016    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:14:57.446294 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:14:57.446202    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:15:01.575259 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.575215    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"]
Oct 31 00:15:01.575814 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.575788    4555 cpu_manager.go:414] "containerMap: removing container" podUID="4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" containerName="main"
Oct 31 00:15:01.575814 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.575808    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" containerName="main"
Oct 31 00:15:01.575947 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.575920    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f52a91f-7d8b-4e5d-8cf4-f27821c8835e" containerName="main"
Oct 31 00:15:01.580289 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.580268    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:01.589214 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.589178    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"]
Oct 31 00:15:01.589789 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.589766    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:15:01.589922 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.589820    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 31 00:15:01.589922 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.589874    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:15:01.589922 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.589907    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:15:01.654042 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.654006    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbcb34bf-xn67x\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:01.654264 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.654049    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5bcf\" (UniqueName: \"kubernetes.io/projected/9a65648d-9530-43f0-8a33-098f6d8d460f-kube-api-access-r5bcf\") pod \"maas-api-revoke-sub-dbcb34bf-xn67x\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:01.654264 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.654208    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbcb34bf-xn67x\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:01.755678 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.755643    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbcb34bf-xn67x\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:01.755902 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.755718    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbcb34bf-xn67x\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:01.755902 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.755751    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5bcf\" (UniqueName: \"kubernetes.io/projected/9a65648d-9530-43f0-8a33-098f6d8d460f-kube-api-access-r5bcf\") pod \"maas-api-revoke-sub-dbcb34bf-xn67x\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:01.756338 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.756313    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbcb34bf-xn67x\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:01.758304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.758279    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbcb34bf-xn67x\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:01.769034 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.769010    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5bcf\" (UniqueName: \"kubernetes.io/projected/9a65648d-9530-43f0-8a33-098f6d8d460f-kube-api-access-r5bcf\") pod \"maas-api-revoke-sub-dbcb34bf-xn67x\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:01.896544 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:01.896462    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:02.178236 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:02.178199    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"]
Oct 31 00:15:02.180298 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:15:02.180271    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a65648d_9530_43f0_8a33_098f6d8d460f.slice/crio-5d5302321bba44dc7ec181e6597da8f3a6a3b98b78812bdb2176db7afc977e17 WatchSource:0}: Error finding container 5d5302321bba44dc7ec181e6597da8f3a6a3b98b78812bdb2176db7afc977e17: Status 404 returned error can't find the container with id 5d5302321bba44dc7ec181e6597da8f3a6a3b98b78812bdb2176db7afc977e17
Oct 31 00:15:02.995295 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:02.995258    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x" event={"ID":"9a65648d-9530-43f0-8a33-098f6d8d460f","Type":"ContainerStarted","Data":"f782f299ce890356aee254f77bf5aab4d99830cb55d4569a86966641a6c9565c"}
Oct 31 00:15:02.995713 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:02.995347    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x" event={"ID":"9a65648d-9530-43f0-8a33-098f6d8d460f","Type":"ContainerStarted","Data":"5d5302321bba44dc7ec181e6597da8f3a6a3b98b78812bdb2176db7afc977e17"}
Oct 31 00:15:03.018270 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:03.018221    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x" podStartSLOduration=2.018209091 podStartE2EDuration="2.018209091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:03.016293358 +0000 UTC m=+3361.942930938" watchObservedRunningTime="2026-10-31 00:15:03.018209091 +0000 UTC m=+3361.944846691"
Oct 31 00:15:04.003144 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:04.003105    4555 generic.go:356] "Generic (PLEG): container finished" podID="9a65648d-9530-43f0-8a33-098f6d8d460f" containerID="f782f299ce890356aee254f77bf5aab4d99830cb55d4569a86966641a6c9565c" exitCode=0
Oct 31 00:15:04.003559 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:04.003180    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x" event={"ID":"9a65648d-9530-43f0-8a33-098f6d8d460f","Type":"ContainerDied","Data":"f782f299ce890356aee254f77bf5aab4d99830cb55d4569a86966641a6c9565c"}
Oct 31 00:15:05.142334 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:05.142303    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:05.289711 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:05.289676    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-openshift-ca\") pod \"9a65648d-9530-43f0-8a33-098f6d8d460f\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") "
Oct 31 00:15:05.289919 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:05.289743    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a65648d-9530-43f0-8a33-098f6d8d460f-kube-api-access-r5bcf\" (UniqueName: \"kubernetes.io/projected/9a65648d-9530-43f0-8a33-098f6d8d460f-kube-api-access-r5bcf\") pod \"9a65648d-9530-43f0-8a33-098f6d8d460f\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") "
Oct 31 00:15:05.289919 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:05.289787    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-xks-ca\") pod \"9a65648d-9530-43f0-8a33-098f6d8d460f\" (UID: \"9a65648d-9530-43f0-8a33-098f6d8d460f\") "
Oct 31 00:15:05.290126 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:05.290100    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-openshift-ca" pod "9a65648d-9530-43f0-8a33-098f6d8d460f" (UID: "9a65648d-9530-43f0-8a33-098f6d8d460f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:05.292093 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:05.292065    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a65648d-9530-43f0-8a33-098f6d8d460f-kube-api-access-r5bcf" pod "9a65648d-9530-43f0-8a33-098f6d8d460f" (UID: "9a65648d-9530-43f0-8a33-098f6d8d460f"). InnerVolumeSpecName "kube-api-access-r5bcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:05.292199 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:05.292139    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-xks-ca" pod "9a65648d-9530-43f0-8a33-098f6d8d460f" (UID: "9a65648d-9530-43f0-8a33-098f6d8d460f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:05.390872 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:05.390778    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:05.390872 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:05.390811    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5bcf\" (UniqueName: \"kubernetes.io/projected/9a65648d-9530-43f0-8a33-098f6d8d460f-kube-api-access-r5bcf\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:05.390872 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:05.390825    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a65648d-9530-43f0-8a33-098f6d8d460f-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:06.014508 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:06.014481    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"
Oct 31 00:15:06.014693 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:06.014485    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x" event={"ID":"9a65648d-9530-43f0-8a33-098f6d8d460f","Type":"ContainerDied","Data":"5d5302321bba44dc7ec181e6597da8f3a6a3b98b78812bdb2176db7afc977e17"}
Oct 31 00:15:06.014693 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:06.014613    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d5302321bba44dc7ec181e6597da8f3a6a3b98b78812bdb2176db7afc977e17"
Oct 31 00:15:12.731228 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:12.731143    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:15:27.447759 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:15:27.447721    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:15:27.766812 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:27.766719    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:15:40.253988 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.253947    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"]
Oct 31 00:15:40.254546 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.254527    4555 cpu_manager.go:414] "containerMap: removing container" podUID="9a65648d-9530-43f0-8a33-098f6d8d460f" containerName="revoke-keys"
Oct 31 00:15:40.254620 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.254548    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a65648d-9530-43f0-8a33-098f6d8d460f" containerName="revoke-keys"
Oct 31 00:15:40.254681 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.254640    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a65648d-9530-43f0-8a33-098f6d8d460f" containerName="revoke-keys"
Oct 31 00:15:40.257335 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.257313    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:40.282229 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.282202    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:15:40.282684 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.282669    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bmrd2\""
Oct 31 00:15:40.299283 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.299257    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"]
Oct 31 00:15:40.307142 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.307116    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjv9\" (UniqueName: \"kubernetes.io/projected/7e85aa9c-468e-482c-a10b-c87476562eb6-kube-api-access-zpjv9\") pod \"maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:40.307303 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.307198    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:40.307463 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.307440    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:40.311461 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.311438    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:15:40.311561 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.311516    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:15:40.408521 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.408487    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:40.408706 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.408537    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:40.408706 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.408581    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjv9\" (UniqueName: \"kubernetes.io/projected/7e85aa9c-468e-482c-a10b-c87476562eb6-kube-api-access-zpjv9\") pod \"maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:40.409207 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.409176    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:40.411065 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:40.411042    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:40.414513 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:15:40.414495    4555 projected.go:196] Error preparing data for projected volume kube-api-access-zpjv9 for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 00:15:40.414630 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:15:40.414570    4555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e85aa9c-468e-482c-a10b-c87476562eb6-kube-api-access-zpjv9 podName:7e85aa9c-468e-482c-a10b-c87476562eb6 nodeName:}" failed. No retries permitted until 2026-10-31 00:15:40.914550403 +0000 UTC m=+3399.841187978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zpjv9" (UniqueName: "kubernetes.io/projected/7e85aa9c-468e-482c-a10b-c87476562eb6-kube-api-access-zpjv9") pod "maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk" (UID: "7e85aa9c-468e-482c-a10b-c87476562eb6") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 00:15:41.014703 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:41.014660    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjv9\" (UniqueName: \"kubernetes.io/projected/7e85aa9c-468e-482c-a10b-c87476562eb6-kube-api-access-zpjv9\") pod \"maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:41.027242 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:41.027213    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjv9\" (UniqueName: \"kubernetes.io/projected/7e85aa9c-468e-482c-a10b-c87476562eb6-kube-api-access-zpjv9\") pod \"maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:41.170258 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:41.170217    4555 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk" secret="" err="secret \"maas-api-cleanup-dockercfg-bmrd2\" not found"
Oct 31 00:15:41.170503 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:41.170271    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:41.451238 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:41.451211    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"]
Oct 31 00:15:41.453125 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:15:41.453096    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e85aa9c_468e_482c_a10b_c87476562eb6.slice/crio-b9a3af479a635434afdcd4345b664f3b747997a7f4a5a4376680d852f93956fd WatchSource:0}: Error finding container b9a3af479a635434afdcd4345b664f3b747997a7f4a5a4376680d852f93956fd: Status 404 returned error can't find the container with id b9a3af479a635434afdcd4345b664f3b747997a7f4a5a4376680d852f93956fd
Oct 31 00:15:42.202707 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:42.202613    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk" event={"ID":"7e85aa9c-468e-482c-a10b-c87476562eb6","Type":"ContainerStarted","Data":"d37c574c97d60fea2821cceaf2bd58c5ad7eed782f62407860ee44be349c5d68"}
Oct 31 00:15:42.202707 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:42.202675    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk" event={"ID":"7e85aa9c-468e-482c-a10b-c87476562eb6","Type":"ContainerStarted","Data":"b9a3af479a635434afdcd4345b664f3b747997a7f4a5a4376680d852f93956fd"}
Oct 31 00:15:42.202976 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:42.202957    4555 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk" secret="" err="secret \"maas-api-cleanup-dockercfg-bmrd2\" not found"
Oct 31 00:15:42.222457 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:42.222399    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk" podStartSLOduration=2.222385777 podStartE2EDuration="2.222385777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:42.220803172 +0000 UTC m=+3401.147440765" watchObservedRunningTime="2026-10-31 00:15:42.222385777 +0000 UTC m=+3401.149023349"
Oct 31 00:15:43.209401 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:43.209351    4555 generic.go:356] "Generic (PLEG): container finished" podID="7e85aa9c-468e-482c-a10b-c87476562eb6" containerID="d37c574c97d60fea2821cceaf2bd58c5ad7eed782f62407860ee44be349c5d68" exitCode=0
Oct 31 00:15:43.209861 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:43.209433    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk" event={"ID":"7e85aa9c-468e-482c-a10b-c87476562eb6","Type":"ContainerDied","Data":"d37c574c97d60fea2821cceaf2bd58c5ad7eed782f62407860ee44be349c5d68"}
Oct 31 00:15:44.350208 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:44.350183    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:44.451589 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:44.451552    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e85aa9c-468e-482c-a10b-c87476562eb6-kube-api-access-zpjv9\" (UniqueName: \"kubernetes.io/projected/7e85aa9c-468e-482c-a10b-c87476562eb6-kube-api-access-zpjv9\") pod \"7e85aa9c-468e-482c-a10b-c87476562eb6\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") "
Oct 31 00:15:44.451806 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:44.451609    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-openshift-ca\") pod \"7e85aa9c-468e-482c-a10b-c87476562eb6\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") "
Oct 31 00:15:44.451806 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:44.451731    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-xks-ca\") pod \"7e85aa9c-468e-482c-a10b-c87476562eb6\" (UID: \"7e85aa9c-468e-482c-a10b-c87476562eb6\") "
Oct 31 00:15:44.452010 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:44.451983    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-openshift-ca" pod "7e85aa9c-468e-482c-a10b-c87476562eb6" (UID: "7e85aa9c-468e-482c-a10b-c87476562eb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:44.453983 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:44.453947    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e85aa9c-468e-482c-a10b-c87476562eb6-kube-api-access-zpjv9" pod "7e85aa9c-468e-482c-a10b-c87476562eb6" (UID: "7e85aa9c-468e-482c-a10b-c87476562eb6"). InnerVolumeSpecName "kube-api-access-zpjv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:44.453983 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:44.453965    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-xks-ca" pod "7e85aa9c-468e-482c-a10b-c87476562eb6" (UID: "7e85aa9c-468e-482c-a10b-c87476562eb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:44.553481 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:44.553446    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:44.553481 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:44.553478    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85aa9c-468e-482c-a10b-c87476562eb6-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:44.553481 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:44.553490    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpjv9\" (UniqueName: \"kubernetes.io/projected/7e85aa9c-468e-482c-a10b-c87476562eb6-kube-api-access-zpjv9\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:45.220410 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:45.220363    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk" event={"ID":"7e85aa9c-468e-482c-a10b-c87476562eb6","Type":"ContainerDied","Data":"b9a3af479a635434afdcd4345b664f3b747997a7f4a5a4376680d852f93956fd"}
Oct 31 00:15:45.220584 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:45.220413    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"
Oct 31 00:15:45.220584 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:45.220433    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9a3af479a635434afdcd4345b664f3b747997a7f4a5a4376680d852f93956fd"
Oct 31 00:15:51.141130 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:51.141093    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"]
Oct 31 00:15:51.148782 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:51.148752    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48ebba5a-lvtpk"]
Oct 31 00:15:51.207815 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:51.207778    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e85aa9c-468e-482c-a10b-c87476562eb6" path="/var/lib/kubelet/pods/7e85aa9c-468e-482c-a10b-c87476562eb6/volumes"
Oct 31 00:15:52.566322 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:15:52.566270    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:15:57.447239 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:15:57.447193    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:16:05.144533 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:16:05.144491    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:16:27.446797 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:16:27.446755    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:16:57.445006 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:16:57.444966    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:17:27.445051 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:17:27.445009    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:17:37.306779 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.306747    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"]
Oct 31 00:17:37.307214 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.307117    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7e85aa9c-468e-482c-a10b-c87476562eb6" containerName="revoke-keys"
Oct 31 00:17:37.307214 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.307129    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e85aa9c-468e-482c-a10b-c87476562eb6" containerName="revoke-keys"
Oct 31 00:17:37.307214 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.307212    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e85aa9c-468e-482c-a10b-c87476562eb6" containerName="revoke-keys"
Oct 31 00:17:37.310025 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.310009    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.316162 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.316138    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:17:37.317442 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.317248    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:17:37.317683 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.317654    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:17:37.317827 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.317803    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkwwb\""
Oct 31 00:17:37.320408 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.320364    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"]
Oct 31 00:17:37.405077 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.405037    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.405262 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.405167    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.405262 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.405207    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvqz\" (UniqueName: \"kubernetes.io/projected/44634230-fc6e-422e-bd92-b10fbb846f82-kube-api-access-cdvqz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.506423 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.506341    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.506640 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.506502    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.506640 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.506545    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvqz\" (UniqueName: \"kubernetes.io/projected/44634230-fc6e-422e-bd92-b10fbb846f82-kube-api-access-cdvqz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.507011 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.506988    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.509239 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.509213    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.519432 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.519410    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvqz\" (UniqueName: \"kubernetes.io/projected/44634230-fc6e-422e-bd92-b10fbb846f82-kube-api-access-cdvqz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.625332 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.625232    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:37.916840 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:37.916808    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"]
Oct 31 00:17:37.918587 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:17:37.918559    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44634230_fc6e_422e_bd92_b10fbb846f82.slice/crio-41f03f298d3fbc072998f481fb3ffc70192b421252219a0225329d2cdda379c2 WatchSource:0}: Error finding container 41f03f298d3fbc072998f481fb3ffc70192b421252219a0225329d2cdda379c2: Status 404 returned error can't find the container with id 41f03f298d3fbc072998f481fb3ffc70192b421252219a0225329d2cdda379c2
Oct 31 00:17:38.831444 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:38.831408    4555 generic.go:356] "Generic (PLEG): container finished" podID="44634230-fc6e-422e-bd92-b10fbb846f82" containerID="2b008b00c18244829eb4cc684f73c9416d010d9d7fb3580dc14727c582701cab" exitCode=0
Oct 31 00:17:38.831826 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:38.831480    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd" event={"ID":"44634230-fc6e-422e-bd92-b10fbb846f82","Type":"ContainerDied","Data":"2b008b00c18244829eb4cc684f73c9416d010d9d7fb3580dc14727c582701cab"}
Oct 31 00:17:38.831826 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:38.831519    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd" event={"ID":"44634230-fc6e-422e-bd92-b10fbb846f82","Type":"ContainerStarted","Data":"41f03f298d3fbc072998f481fb3ffc70192b421252219a0225329d2cdda379c2"}
Oct 31 00:17:39.971162 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:39.971135    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:40.030748 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.030711    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-openshift-ca\") pod \"44634230-fc6e-422e-bd92-b10fbb846f82\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") "
Oct 31 00:17:40.030943 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.030830    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44634230-fc6e-422e-bd92-b10fbb846f82-kube-api-access-cdvqz\" (UniqueName: \"kubernetes.io/projected/44634230-fc6e-422e-bd92-b10fbb846f82-kube-api-access-cdvqz\") pod \"44634230-fc6e-422e-bd92-b10fbb846f82\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") "
Oct 31 00:17:40.030943 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.030867    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-xks-ca\") pod \"44634230-fc6e-422e-bd92-b10fbb846f82\" (UID: \"44634230-fc6e-422e-bd92-b10fbb846f82\") "
Oct 31 00:17:40.031081 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.031065    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-openshift-ca" pod "44634230-fc6e-422e-bd92-b10fbb846f82" (UID: "44634230-fc6e-422e-bd92-b10fbb846f82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:40.033061 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.033033    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44634230-fc6e-422e-bd92-b10fbb846f82-kube-api-access-cdvqz" pod "44634230-fc6e-422e-bd92-b10fbb846f82" (UID: "44634230-fc6e-422e-bd92-b10fbb846f82"). InnerVolumeSpecName "kube-api-access-cdvqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:40.033176 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.033113    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-xks-ca" pod "44634230-fc6e-422e-bd92-b10fbb846f82" (UID: "44634230-fc6e-422e-bd92-b10fbb846f82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:40.131977 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.131884    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:40.131977 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.131916    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdvqz\" (UniqueName: \"kubernetes.io/projected/44634230-fc6e-422e-bd92-b10fbb846f82-kube-api-access-cdvqz\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:40.131977 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.131932    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44634230-fc6e-422e-bd92-b10fbb846f82-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:40.842958 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.842918    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd" event={"ID":"44634230-fc6e-422e-bd92-b10fbb846f82","Type":"ContainerDied","Data":"41f03f298d3fbc072998f481fb3ffc70192b421252219a0225329d2cdda379c2"}
Oct 31 00:17:40.842958 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.842939    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"
Oct 31 00:17:40.843180 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:40.842970    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41f03f298d3fbc072998f481fb3ffc70192b421252219a0225329d2cdda379c2"
Oct 31 00:17:53.130756 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:53.130719    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"]
Oct 31 00:17:53.132942 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:53.132910    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b33f34-9ncxd"]
Oct 31 00:17:53.213561 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:53.213512    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44634230-fc6e-422e-bd92-b10fbb846f82" path="/var/lib/kubelet/pods/44634230-fc6e-422e-bd92-b10fbb846f82/volumes"
Oct 31 00:17:54.473265 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:17:54.473231    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:17:57.446346 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:17:57.446309    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:18:02.230570 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:02.230536    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:18:07.344469 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:07.344434    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:18:11.562772 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.562683    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"]
Oct 31 00:18:11.563243 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.563219    4555 cpu_manager.go:414] "containerMap: removing container" podUID="44634230-fc6e-422e-bd92-b10fbb846f82" containerName="revoke-keys"
Oct 31 00:18:11.563243 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.563238    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44634230-fc6e-422e-bd92-b10fbb846f82" containerName="revoke-keys"
Oct 31 00:18:11.563396 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.563364    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="44634230-fc6e-422e-bd92-b10fbb846f82" containerName="revoke-keys"
Oct 31 00:18:11.566762 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.566743    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:11.574102 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.573590    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:11.574102 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.573648    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:18:11.574102 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.573674    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkwwb\""
Oct 31 00:18:11.574102 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.573590    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:18:11.577768 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.577747    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"]
Oct 31 00:18:11.674917 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.674882    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjdjv\" (UniqueName: \"kubernetes.io/projected/3de3d2b7-9ba2-4b14-954a-12b008a46d46-kube-api-access-wjdjv\") pod \"maas-api-revoke-sub-f03629db-56wnb\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:11.674917 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.674922    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-56wnb\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:11.675155 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.674942    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-56wnb\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:11.775957 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.775916    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjdjv\" (UniqueName: \"kubernetes.io/projected/3de3d2b7-9ba2-4b14-954a-12b008a46d46-kube-api-access-wjdjv\") pod \"maas-api-revoke-sub-f03629db-56wnb\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:11.776158 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.775962    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-56wnb\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:11.776158 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.775991    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-56wnb\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:11.776711 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.776686    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-56wnb\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:11.778433 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.778411    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-56wnb\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:11.788720 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.788689    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjdjv\" (UniqueName: \"kubernetes.io/projected/3de3d2b7-9ba2-4b14-954a-12b008a46d46-kube-api-access-wjdjv\") pod \"maas-api-revoke-sub-f03629db-56wnb\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:11.881410 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:11.881316    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:12.159169 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:12.159145    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"]
Oct 31 00:18:12.161415 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:18:12.161355    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de3d2b7_9ba2_4b14_954a_12b008a46d46.slice/crio-92bd03c9814fb8fff446338eafcb607a3bb7ef5215dd161579b14b645486a108 WatchSource:0}: Error finding container 92bd03c9814fb8fff446338eafcb607a3bb7ef5215dd161579b14b645486a108: Status 404 returned error can't find the container with id 92bd03c9814fb8fff446338eafcb607a3bb7ef5215dd161579b14b645486a108
Oct 31 00:18:13.046139 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:13.046108    4555 generic.go:356] "Generic (PLEG): container finished" podID="3de3d2b7-9ba2-4b14-954a-12b008a46d46" containerID="c5453788311006621b3f5199b4fd65bcd901959686c17db66fb9a38050548bb0" exitCode=0
Oct 31 00:18:13.046529 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:13.046188    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb" event={"ID":"3de3d2b7-9ba2-4b14-954a-12b008a46d46","Type":"ContainerDied","Data":"c5453788311006621b3f5199b4fd65bcd901959686c17db66fb9a38050548bb0"}
Oct 31 00:18:13.046529 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:13.046248    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb" event={"ID":"3de3d2b7-9ba2-4b14-954a-12b008a46d46","Type":"ContainerStarted","Data":"92bd03c9814fb8fff446338eafcb607a3bb7ef5215dd161579b14b645486a108"}
Oct 31 00:18:14.188515 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:14.188491    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:14.297491 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:14.297454    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-openshift-ca\") pod \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") "
Oct 31 00:18:14.297685 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:14.297528    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-xks-ca\") pod \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") "
Oct 31 00:18:14.297685 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:14.297586    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de3d2b7-9ba2-4b14-954a-12b008a46d46-kube-api-access-wjdjv\" (UniqueName: \"kubernetes.io/projected/3de3d2b7-9ba2-4b14-954a-12b008a46d46-kube-api-access-wjdjv\") pod \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\" (UID: \"3de3d2b7-9ba2-4b14-954a-12b008a46d46\") "
Oct 31 00:18:14.297904 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:14.297872    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-openshift-ca" pod "3de3d2b7-9ba2-4b14-954a-12b008a46d46" (UID: "3de3d2b7-9ba2-4b14-954a-12b008a46d46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:14.299767 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:14.299741    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-xks-ca" pod "3de3d2b7-9ba2-4b14-954a-12b008a46d46" (UID: "3de3d2b7-9ba2-4b14-954a-12b008a46d46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:14.299890 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:14.299877    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de3d2b7-9ba2-4b14-954a-12b008a46d46-kube-api-access-wjdjv" pod "3de3d2b7-9ba2-4b14-954a-12b008a46d46" (UID: "3de3d2b7-9ba2-4b14-954a-12b008a46d46"). InnerVolumeSpecName "kube-api-access-wjdjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:14.398753 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:14.398712    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjdjv\" (UniqueName: \"kubernetes.io/projected/3de3d2b7-9ba2-4b14-954a-12b008a46d46-kube-api-access-wjdjv\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:14.398753 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:14.398748    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:14.398753 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:14.398758    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de3d2b7-9ba2-4b14-954a-12b008a46d46-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:15.057691 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:15.057659    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb"
Oct 31 00:18:15.057863 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:15.057695    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-56wnb" event={"ID":"3de3d2b7-9ba2-4b14-954a-12b008a46d46","Type":"ContainerDied","Data":"92bd03c9814fb8fff446338eafcb607a3bb7ef5215dd161579b14b645486a108"}
Oct 31 00:18:15.057863 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:15.057747    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92bd03c9814fb8fff446338eafcb607a3bb7ef5215dd161579b14b645486a108"
Oct 31 00:18:22.425843 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:22.425801    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:18:27.289124 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.289089    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"]
Oct 31 00:18:27.289537 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.289469    4555 cpu_manager.go:414] "containerMap: removing container" podUID="3de3d2b7-9ba2-4b14-954a-12b008a46d46" containerName="revoke-keys"
Oct 31 00:18:27.289537 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.289481    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de3d2b7-9ba2-4b14-954a-12b008a46d46" containerName="revoke-keys"
Oct 31 00:18:27.289612 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.289546    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de3d2b7-9ba2-4b14-954a-12b008a46d46" containerName="revoke-keys"
Oct 31 00:18:27.295552 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.293625    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.299162 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.299139    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:27.300637 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.300613    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:18:27.300637 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.300621    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:18:27.300847 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.300658    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkwwb\""
Oct 31 00:18:27.302526 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.302501    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"]
Oct 31 00:18:27.444673 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.444638    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bxg84\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.444855 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.444703    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wp6\" (UniqueName: \"kubernetes.io/projected/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-kube-api-access-g2wp6\") pod \"maas-api-revoke-sub-0165d7d2-bxg84\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.444855 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.444802    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bxg84\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.446113 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:18:27.446084    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:18:27.545792 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.545694    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bxg84\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.545792 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.545748    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wp6\" (UniqueName: \"kubernetes.io/projected/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-kube-api-access-g2wp6\") pod \"maas-api-revoke-sub-0165d7d2-bxg84\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.545792 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.545778    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bxg84\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.546511 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.546491    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bxg84\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.548273 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.548248    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bxg84\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.558331 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.558309    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wp6\" (UniqueName: \"kubernetes.io/projected/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-kube-api-access-g2wp6\") pod \"maas-api-revoke-sub-0165d7d2-bxg84\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.613220 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.613180    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:27.891104 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:27.890977    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"]
Oct 31 00:18:27.893932 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:18:27.893900    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf0585f_0356_426f_b4e0_bd8a3a1f5492.slice/crio-f65057b84143a1bb3e39623ad41bc197e8afd2be4823090b375f88ed804ec50d WatchSource:0}: Error finding container f65057b84143a1bb3e39623ad41bc197e8afd2be4823090b375f88ed804ec50d: Status 404 returned error can't find the container with id f65057b84143a1bb3e39623ad41bc197e8afd2be4823090b375f88ed804ec50d
Oct 31 00:18:28.134981 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:28.134948    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84" event={"ID":"aaf0585f-0356-426f-b4e0-bd8a3a1f5492","Type":"ContainerStarted","Data":"f65057b84143a1bb3e39623ad41bc197e8afd2be4823090b375f88ed804ec50d"}
Oct 31 00:18:29.141524 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:29.141485    4555 generic.go:356] "Generic (PLEG): container finished" podID="aaf0585f-0356-426f-b4e0-bd8a3a1f5492" containerID="8b4aeea59fd1d9f35b90d5433e1c37c4acdf5725a03c7b667a7747c0dbd50388" exitCode=0
Oct 31 00:18:29.141992 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:29.141531    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84" event={"ID":"aaf0585f-0356-426f-b4e0-bd8a3a1f5492","Type":"ContainerDied","Data":"8b4aeea59fd1d9f35b90d5433e1c37c4acdf5725a03c7b667a7747c0dbd50388"}
Oct 31 00:18:30.280864 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:30.280836    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:30.371813 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:30.371778    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-openshift-ca\") pod \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") "
Oct 31 00:18:30.371994 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:30.371824    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-kube-api-access-g2wp6\" (UniqueName: \"kubernetes.io/projected/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-kube-api-access-g2wp6\") pod \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") "
Oct 31 00:18:30.371994 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:30.371866    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-xks-ca\") pod \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\" (UID: \"aaf0585f-0356-426f-b4e0-bd8a3a1f5492\") "
Oct 31 00:18:30.372231 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:30.372207    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-openshift-ca" pod "aaf0585f-0356-426f-b4e0-bd8a3a1f5492" (UID: "aaf0585f-0356-426f-b4e0-bd8a3a1f5492"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:30.374089 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:30.374069    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-xks-ca" pod "aaf0585f-0356-426f-b4e0-bd8a3a1f5492" (UID: "aaf0585f-0356-426f-b4e0-bd8a3a1f5492"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:30.374168 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:30.374148    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-kube-api-access-g2wp6" pod "aaf0585f-0356-426f-b4e0-bd8a3a1f5492" (UID: "aaf0585f-0356-426f-b4e0-bd8a3a1f5492"). InnerVolumeSpecName "kube-api-access-g2wp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:30.472737 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:30.472664    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2wp6\" (UniqueName: \"kubernetes.io/projected/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-kube-api-access-g2wp6\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:30.472737 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:30.472690    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:30.472737 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:30.472700    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf0585f-0356-426f-b4e0-bd8a3a1f5492-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:31.153775 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:31.153739    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84" event={"ID":"aaf0585f-0356-426f-b4e0-bd8a3a1f5492","Type":"ContainerDied","Data":"f65057b84143a1bb3e39623ad41bc197e8afd2be4823090b375f88ed804ec50d"}
Oct 31 00:18:31.153959 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:31.153779    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bxg84"
Oct 31 00:18:31.153959 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:31.153791    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f65057b84143a1bb3e39623ad41bc197e8afd2be4823090b375f88ed804ec50d"
Oct 31 00:18:38.137751 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.137708    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"]
Oct 31 00:18:38.138303 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.138286    4555 cpu_manager.go:414] "containerMap: removing container" podUID="aaf0585f-0356-426f-b4e0-bd8a3a1f5492" containerName="revoke-keys"
Oct 31 00:18:38.138349 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.138306    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf0585f-0356-426f-b4e0-bd8a3a1f5492" containerName="revoke-keys"
Oct 31 00:18:38.138460 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.138447    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf0585f-0356-426f-b4e0-bd8a3a1f5492" containerName="revoke-keys"
Oct 31 00:18:38.141753 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.141728    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.148053 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.147886    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:18:38.148053 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.147893    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:38.148053 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.147958    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:18:38.148053 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.148023    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkwwb\""
Oct 31 00:18:38.151234 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.151205    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"]
Oct 31 00:18:38.237276 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.237226    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9h2j9\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.237516 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.237396    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9h2j9\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.237516 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.237429    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qxc7\" (UniqueName: \"kubernetes.io/projected/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-kube-api-access-7qxc7\") pod \"maas-api-revoke-sub-6b19338f-9h2j9\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.338605 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.338566    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9h2j9\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.338820 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.338656    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9h2j9\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.338820 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.338686    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qxc7\" (UniqueName: \"kubernetes.io/projected/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-kube-api-access-7qxc7\") pod \"maas-api-revoke-sub-6b19338f-9h2j9\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.339225 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.339202    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9h2j9\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.341183 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.341161    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9h2j9\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.351899 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.351877    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qxc7\" (UniqueName: \"kubernetes.io/projected/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-kube-api-access-7qxc7\") pod \"maas-api-revoke-sub-6b19338f-9h2j9\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.459338 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.459251    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:38.741429 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:38.741402    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"]
Oct 31 00:18:38.744033 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:18:38.744003    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba40eaa_9477_4b78_9e59_e69b6f6dde3c.slice/crio-7f80c870c7f6fb340873f9d48fa0675e7996dd064adee038f6bdd08eca5ce235 WatchSource:0}: Error finding container 7f80c870c7f6fb340873f9d48fa0675e7996dd064adee038f6bdd08eca5ce235: Status 404 returned error can't find the container with id 7f80c870c7f6fb340873f9d48fa0675e7996dd064adee038f6bdd08eca5ce235
Oct 31 00:18:39.199258 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:39.199221    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9" event={"ID":"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c","Type":"ContainerStarted","Data":"1f383b00c60f7431bc448688e0cb05a09d9901c0af68bac6d98de08f8607744a"}
Oct 31 00:18:39.199761 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:39.199291    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9" event={"ID":"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c","Type":"ContainerStarted","Data":"7f80c870c7f6fb340873f9d48fa0675e7996dd064adee038f6bdd08eca5ce235"}
Oct 31 00:18:39.219864 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:39.219817    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9" podStartSLOduration=1.21980312 podStartE2EDuration="1.21980312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:39.217337007 +0000 UTC m=+3578.143974587" watchObservedRunningTime="2026-10-31 00:18:39.21980312 +0000 UTC m=+3578.146440700"
Oct 31 00:18:40.206264 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:40.206223    4555 generic.go:356] "Generic (PLEG): container finished" podID="7ba40eaa-9477-4b78-9e59-e69b6f6dde3c" containerID="1f383b00c60f7431bc448688e0cb05a09d9901c0af68bac6d98de08f8607744a" exitCode=0
Oct 31 00:18:40.206697 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:40.206318    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9" event={"ID":"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c","Type":"ContainerDied","Data":"1f383b00c60f7431bc448688e0cb05a09d9901c0af68bac6d98de08f8607744a"}
Oct 31 00:18:41.340709 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:41.340682    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:41.367887 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:41.367855    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-openshift-ca\") pod \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") "
Oct 31 00:18:41.368068 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:41.367903    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-xks-ca\") pod \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") "
Oct 31 00:18:41.368068 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:41.367942    4555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-kube-api-access-7qxc7\" (UniqueName: \"kubernetes.io/projected/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-kube-api-access-7qxc7\") pod \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\" (UID: \"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c\") "
Oct 31 00:18:41.368266 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:41.368237    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-openshift-ca" pod "7ba40eaa-9477-4b78-9e59-e69b6f6dde3c" (UID: "7ba40eaa-9477-4b78-9e59-e69b6f6dde3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:41.370515 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:41.370483    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-xks-ca" pod "7ba40eaa-9477-4b78-9e59-e69b6f6dde3c" (UID: "7ba40eaa-9477-4b78-9e59-e69b6f6dde3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:41.370515 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:41.370492    4555 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-kube-api-access-7qxc7" pod "7ba40eaa-9477-4b78-9e59-e69b6f6dde3c" (UID: "7ba40eaa-9477-4b78-9e59-e69b6f6dde3c"). InnerVolumeSpecName "kube-api-access-7qxc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:41.468874 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:41.468787    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:41.468874 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:41.468825    4555 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:41.468874 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:41.468835    4555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qxc7\" (UniqueName: \"kubernetes.io/projected/7ba40eaa-9477-4b78-9e59-e69b6f6dde3c-kube-api-access-7qxc7\") on node \"ip-10-0-134-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:42.217430 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:42.217401    4555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9"
Oct 31 00:18:42.217639 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:42.217437    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9h2j9" event={"ID":"7ba40eaa-9477-4b78-9e59-e69b6f6dde3c","Type":"ContainerDied","Data":"7f80c870c7f6fb340873f9d48fa0675e7996dd064adee038f6bdd08eca5ce235"}
Oct 31 00:18:42.217639 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:42.217489    4555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f80c870c7f6fb340873f9d48fa0675e7996dd064adee038f6bdd08eca5ce235"
Oct 31 00:18:49.236017 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:49.235976    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9w98b"]
Oct 31 00:18:51.017397 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:51.016492    4555 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dlhf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:18:51.017397 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:18:51.016580    4555 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dlhf9" podUID="cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:18:57.445614 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:18:57.445567    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:19:00.772044 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:00.772010    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pm8s9_78b47baf-3103-4d30-953e-62628d4a795c/manager/0.log"
Oct 31 00:19:01.182939 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:01.182866    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qpd89_d4e50936-fbfc-49ab-b816-30d80f7dc3cb/manager/0.log"
Oct 31 00:19:03.681122 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:03.681085    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-bxg84_aaf0585f-0356-426f-b4e0-bd8a3a1f5492/revoke-keys/0.log"
Oct 31 00:19:04.216488 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:04.216451    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9h2j9_7ba40eaa-9477-4b78-9e59-e69b6f6dde3c/revoke-keys/0.log"
Oct 31 00:19:04.389015 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:04.388983    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dbcb34bf-xn67x_9a65648d-9530-43f0-8a33-098f6d8d460f/revoke-keys/0.log"
Oct 31 00:19:04.578229 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:04.578200    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-56wnb_3de3d2b7-9ba2-4b14-954a-12b008a46d46/revoke-keys/0.log"
Oct 31 00:19:06.569456 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:06.569422    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-fjn4c_67c3b2ff-66e7-466f-a332-8e6b96ec54cf/manager/0.log"
Oct 31 00:19:06.776701 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:06.776669    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5fdkn_e28503fb-c336-451f-a3a0-bd3e8d0673a3/manager/0.log"
Oct 31 00:19:06.958633 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:06.958560    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wfl4g_f87feb72-91e5-456e-8035-68fed3fa7583/kuadrant-console-plugin/0.log"
Oct 31 00:19:07.165955 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:07.165921    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tzfff_3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54/manager/0.log"
Oct 31 00:19:07.415668 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:07.415638    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9w98b_6f402bfd-be3a-4b13-b402-4e6b75dfe2bd/limitador/0.log"
Oct 31 00:19:07.602582 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:07.602550    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x4x9r_bb713bda-562e-4214-976d-1bac69ba3efd/manager/0.log"
Oct 31 00:19:08.390910 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:08.390880    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f859hrmz_91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2/istio-proxy/0.log"
Oct 31 00:19:09.372629 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:09.372581    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dghms_330a67ac-e5a9-4bb7-9d70-89bfd52a5efe/istio-proxy/0.log"
Oct 31 00:19:09.552521 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:09.552496    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dlhf9_cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d/istio-proxy/0.log"
Oct 31 00:19:10.329004 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:10.328968    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-mjk4b_ea819a48-154f-4c7a-a747-3306ac423289/istio-proxy/1.log"
Oct 31 00:19:11.074639 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:11.074613    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78_f308b264-593d-49b7-b1b9-03c09e9cee62/istio-proxy/0.log"
Oct 31 00:19:12.081318 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:12.081286    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-mjk4b_ea819a48-154f-4c7a-a747-3306ac423289/istio-proxy/0.log"
Oct 31 00:19:24.527537 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:24.527500    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pm8s9_78b47baf-3103-4d30-953e-62628d4a795c/manager/0.log"
Oct 31 00:19:24.906744 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:24.906660    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qpd89_d4e50936-fbfc-49ab-b816-30d80f7dc3cb/manager/0.log"
Oct 31 00:19:26.754014 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:26.753980    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-bxg84_aaf0585f-0356-426f-b4e0-bd8a3a1f5492/revoke-keys/0.log"
Oct 31 00:19:27.292498 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:27.292467    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9h2j9_7ba40eaa-9477-4b78-9e59-e69b6f6dde3c/revoke-keys/0.log"
Oct 31 00:19:27.445048 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:19:27.445013    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:19:27.466634 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:27.466603    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dbcb34bf-xn67x_9a65648d-9530-43f0-8a33-098f6d8d460f/revoke-keys/0.log"
Oct 31 00:19:27.644719 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:27.644648    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-56wnb_3de3d2b7-9ba2-4b14-954a-12b008a46d46/revoke-keys/0.log"
Oct 31 00:19:29.187174 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:29.187135    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-fjn4c_67c3b2ff-66e7-466f-a332-8e6b96ec54cf/manager/0.log"
Oct 31 00:19:29.391106 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:29.391076    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5fdkn_e28503fb-c336-451f-a3a0-bd3e8d0673a3/manager/0.log"
Oct 31 00:19:29.596611 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:29.596578    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wfl4g_f87feb72-91e5-456e-8035-68fed3fa7583/kuadrant-console-plugin/0.log"
Oct 31 00:19:29.790432 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:29.790393    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tzfff_3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54/manager/0.log"
Oct 31 00:19:30.003332 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:30.003254    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9w98b_6f402bfd-be3a-4b13-b402-4e6b75dfe2bd/limitador/0.log"
Oct 31 00:19:30.197456 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:30.197424    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x4x9r_bb713bda-562e-4214-976d-1bac69ba3efd/manager/0.log"
Oct 31 00:19:30.774584 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:30.774559    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f859hrmz_91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2/istio-proxy/0.log"
Oct 31 00:19:31.731988 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:31.731952    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dghms_330a67ac-e5a9-4bb7-9d70-89bfd52a5efe/istio-proxy/0.log"
Oct 31 00:19:31.918774 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:31.918747    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dlhf9_cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d/istio-proxy/0.log"
Oct 31 00:19:32.675825 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:32.675791    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-mjk4b_ea819a48-154f-4c7a-a747-3306ac423289/istio-proxy/1.log"
Oct 31 00:19:33.436928 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:19:33.436892    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78_f308b264-593d-49b7-b1b9-03c09e9cee62/istio-proxy/0.log"
Oct 31 00:19:57.446512 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:19:57.446471    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:20:06.024084 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:20:06.024047    4555 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"]
Oct 31 00:20:06.026277 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:20:06.026249    4555 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbcb34bf-xn67x"]
Oct 31 00:20:07.205852 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:20:07.205813    4555 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a65648d-9530-43f0-8a33-098f6d8d460f" path="/var/lib/kubelet/pods/9a65648d-9530-43f0-8a33-098f6d8d460f/volumes"
Oct 31 00:20:27.445174 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:20:27.445132    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:20:57.445134 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:20:57.445093    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:21:04.874823 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:04.874791    4555 scope.go:122] "RemoveContainer" containerID="f782f299ce890356aee254f77bf5aab4d99830cb55d4569a86966641a6c9565c"
Oct 31 00:21:05.249743 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.249668    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrd6t/must-gather-dvtv4"]
Oct 31 00:21:05.250145 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.250126    4555 cpu_manager.go:414] "containerMap: removing container" podUID="7ba40eaa-9477-4b78-9e59-e69b6f6dde3c" containerName="revoke-keys"
Oct 31 00:21:05.250251 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.250146    4555 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba40eaa-9477-4b78-9e59-e69b6f6dde3c" containerName="revoke-keys"
Oct 31 00:21:05.250251 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.250211    4555 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba40eaa-9477-4b78-9e59-e69b6f6dde3c" containerName="revoke-keys"
Oct 31 00:21:05.254653 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.254632    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrd6t/must-gather-dvtv4"
Oct 31 00:21:05.264417 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.264399    4555 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lrd6t\"/\"default-dockercfg-7c4gn\""
Oct 31 00:21:05.267440 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.267420    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrd6t\"/\"openshift-service-ca.crt\""
Oct 31 00:21:05.267546 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.267448    4555 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrd6t\"/\"kube-root-ca.crt\""
Oct 31 00:21:05.287904 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.287878    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b719e82f-0525-4502-8d75-40dbd1740b74-must-gather-output\") pod \"must-gather-dvtv4\" (UID: \"b719e82f-0525-4502-8d75-40dbd1740b74\") " pod="openshift-must-gather-lrd6t/must-gather-dvtv4"
Oct 31 00:21:05.288019 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.287951    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7q8d\" (UniqueName: \"kubernetes.io/projected/b719e82f-0525-4502-8d75-40dbd1740b74-kube-api-access-h7q8d\") pod \"must-gather-dvtv4\" (UID: \"b719e82f-0525-4502-8d75-40dbd1740b74\") " pod="openshift-must-gather-lrd6t/must-gather-dvtv4"
Oct 31 00:21:05.303582 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.303559    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrd6t/must-gather-dvtv4"]
Oct 31 00:21:05.389071 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.389035    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7q8d\" (UniqueName: \"kubernetes.io/projected/b719e82f-0525-4502-8d75-40dbd1740b74-kube-api-access-h7q8d\") pod \"must-gather-dvtv4\" (UID: \"b719e82f-0525-4502-8d75-40dbd1740b74\") " pod="openshift-must-gather-lrd6t/must-gather-dvtv4"
Oct 31 00:21:05.389296 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.389116    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b719e82f-0525-4502-8d75-40dbd1740b74-must-gather-output\") pod \"must-gather-dvtv4\" (UID: \"b719e82f-0525-4502-8d75-40dbd1740b74\") " pod="openshift-must-gather-lrd6t/must-gather-dvtv4"
Oct 31 00:21:05.389479 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.389460    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b719e82f-0525-4502-8d75-40dbd1740b74-must-gather-output\") pod \"must-gather-dvtv4\" (UID: \"b719e82f-0525-4502-8d75-40dbd1740b74\") " pod="openshift-must-gather-lrd6t/must-gather-dvtv4"
Oct 31 00:21:05.403111 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.403079    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7q8d\" (UniqueName: \"kubernetes.io/projected/b719e82f-0525-4502-8d75-40dbd1740b74-kube-api-access-h7q8d\") pod \"must-gather-dvtv4\" (UID: \"b719e82f-0525-4502-8d75-40dbd1740b74\") " pod="openshift-must-gather-lrd6t/must-gather-dvtv4"
Oct 31 00:21:05.569179 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.569142    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrd6t/must-gather-dvtv4"
Oct 31 00:21:05.854047 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.854019    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrd6t/must-gather-dvtv4"]
Oct 31 00:21:05.855621 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:21:05.855592    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb719e82f_0525_4502_8d75_40dbd1740b74.slice/crio-f005204342ad5831aa3cb5175960c3d07ce86e84ac0ea538350d52ab7820cdd5 WatchSource:0}: Error finding container f005204342ad5831aa3cb5175960c3d07ce86e84ac0ea538350d52ab7820cdd5: Status 404 returned error can't find the container with id f005204342ad5831aa3cb5175960c3d07ce86e84ac0ea538350d52ab7820cdd5
Oct 31 00:21:05.857361 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:05.857344    4555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:21:06.007027 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:06.006995    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrd6t/must-gather-dvtv4" event={"ID":"b719e82f-0525-4502-8d75-40dbd1740b74","Type":"ContainerStarted","Data":"f005204342ad5831aa3cb5175960c3d07ce86e84ac0ea538350d52ab7820cdd5"}
Oct 31 00:21:07.016075 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:07.015979    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrd6t/must-gather-dvtv4" event={"ID":"b719e82f-0525-4502-8d75-40dbd1740b74","Type":"ContainerStarted","Data":"12b859d4708d42d6bbbdd174a100f7efe35cf9168e254b2b1d9617646f99f67d"}
Oct 31 00:21:07.016075 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:07.016033    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrd6t/must-gather-dvtv4" event={"ID":"b719e82f-0525-4502-8d75-40dbd1740b74","Type":"ContainerStarted","Data":"b165c4d53e4e683ba7247f921b1a24806cad857d53d9f264a62d6905fc9b84b0"}
Oct 31 00:21:07.036904 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:07.036852    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lrd6t/must-gather-dvtv4" podStartSLOduration=1.202094209 podStartE2EDuration="2.036839341s" totalImagesPullingTime="834.745132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:07.034087244 +0000 UTC m=+3725.960724826" watchObservedRunningTime="2026-10-31 00:21:07.036839341 +0000 UTC m=+3725.963476921"
Oct 31 00:21:08.983349 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:08.983321    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7g6g7_a6a30dfd-0560-4e49-9890-07cb7f8c8bcd/global-pull-secret-syncer/0.log"
Oct 31 00:21:08.984245 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:08.984221    4555 ???:1] "http: TLS handshake error from 10.0.134.83:33604: EOF"
Oct 31 00:21:09.354949 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:09.354917    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l94rx_321dcdee-cb90-4de4-8c18-39120efda8e4/konnectivity-agent/0.log"
Oct 31 00:21:09.382151 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:09.382115    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-83.us-east-2.compute.internal_17ddb1b82b212964d7163812b102551c/haproxy/0.log"
Oct 31 00:21:15.985684 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:15.985656    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-fjn4c_67c3b2ff-66e7-466f-a332-8e6b96ec54cf/manager/0.log"
Oct 31 00:21:16.023525 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:16.022572    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5fdkn_e28503fb-c336-451f-a3a0-bd3e8d0673a3/manager/0.log"
Oct 31 00:21:16.091618 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:16.091581    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wfl4g_f87feb72-91e5-456e-8035-68fed3fa7583/kuadrant-console-plugin/0.log"
Oct 31 00:21:16.185806 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:16.185670    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tzfff_3eb30b4a-dc12-4cbc-be11-bfcf8ab79b54/manager/0.log"
Oct 31 00:21:16.225662 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:16.225575    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9w98b_6f402bfd-be3a-4b13-b402-4e6b75dfe2bd/limitador/0.log"
Oct 31 00:21:16.325566 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:16.325530    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x4x9r_bb713bda-562e-4214-976d-1bac69ba3efd/manager/0.log"
Oct 31 00:21:18.219201 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.219162    4555 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"]
Oct 31 00:21:18.226217 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.226184    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.234269 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.234236    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"]
Oct 31 00:21:18.339551 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.339521    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-lib-modules\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.339749 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.339595    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-sys\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.339749 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.339699    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wml5\" (UniqueName: \"kubernetes.io/projected/5b6095da-47b0-449d-958d-e075b8d68be8-kube-api-access-4wml5\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.339749 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.339742    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-proc\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.339933 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.339825    4555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-podres\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.441193 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.441154    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-lib-modules\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.441413 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.441244    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-sys\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.441413 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.441275    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wml5\" (UniqueName: \"kubernetes.io/projected/5b6095da-47b0-449d-958d-e075b8d68be8-kube-api-access-4wml5\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.441413 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.441300    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-proc\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.441413 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.441312    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-lib-modules\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.441413 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.441325    4555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-podres\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.441413 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.441390    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-podres\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.441413 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.441391    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-sys\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.441855 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.441432    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b6095da-47b0-449d-958d-e075b8d68be8-proc\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.455770 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.455740    4555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wml5\" (UniqueName: \"kubernetes.io/projected/5b6095da-47b0-449d-958d-e075b8d68be8-kube-api-access-4wml5\") pod \"perf-node-gather-daemonset-qdgkm\" (UID: \"5b6095da-47b0-449d-958d-e075b8d68be8\") " pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.460304 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.460279    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zjgtn_3b05378f-b90e-47f8-9958-20da20c16505/node-exporter/0.log"
Oct 31 00:21:18.488969 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.488893    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zjgtn_3b05378f-b90e-47f8-9958-20da20c16505/kube-rbac-proxy/0.log"
Oct 31 00:21:18.517892 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.517862    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zjgtn_3b05378f-b90e-47f8-9958-20da20c16505/init-textfile/0.log"
Oct 31 00:21:18.544624 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.544594    4555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:18.845508 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.845474    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8k4pb_14801d31-6150-4b26-8429-b5a66d372117/prometheus-operator/0.log"
Oct 31 00:21:18.850903 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.850872    4555 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"]
Oct 31 00:21:18.852468 ip-10-0-134-83 kubenswrapper[4555]: W1031 00:21:18.852434    4555 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5b6095da_47b0_449d_958d_e075b8d68be8.slice/crio-ba495fc2c1ceb6e597ee320bb2849dd8189fc10edac762a1eb41e0bb825a64d6 WatchSource:0}: Error finding container ba495fc2c1ceb6e597ee320bb2849dd8189fc10edac762a1eb41e0bb825a64d6: Status 404 returned error can't find the container with id ba495fc2c1ceb6e597ee320bb2849dd8189fc10edac762a1eb41e0bb825a64d6
Oct 31 00:21:18.870912 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.870886    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8k4pb_14801d31-6150-4b26-8429-b5a66d372117/kube-rbac-proxy/0.log"
Oct 31 00:21:18.899305 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:18.899279    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-74csp_5dfa02be-1cdd-462a-bdec-b5c0c182e1f4/prometheus-operator-admission-webhook/0.log"
Oct 31 00:21:19.157287 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:19.157211    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm" event={"ID":"5b6095da-47b0-449d-958d-e075b8d68be8","Type":"ContainerStarted","Data":"7381de98fc7d24b1720d34a7549604209d35583c1d1225eaeb4dbd9f0b9b98f7"}
Oct 31 00:21:19.157287 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:19.157262    4555 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm" event={"ID":"5b6095da-47b0-449d-958d-e075b8d68be8","Type":"ContainerStarted","Data":"ba495fc2c1ceb6e597ee320bb2849dd8189fc10edac762a1eb41e0bb825a64d6"}
Oct 31 00:21:19.157569 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:19.157396    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:19.178978 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:19.178913    4555 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm" podStartSLOduration=1.1788962490000001 podStartE2EDuration="1.178896249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:19.178722905 +0000 UTC m=+3738.105360487" watchObservedRunningTime="2026-10-31 00:21:19.178896249 +0000 UTC m=+3738.105533832"
Oct 31 00:21:21.481927 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:21.481894    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85db5d76c9-xddv6_83ca2764-5d6e-4fbe-9af9-eb80bdf946b3/console/0.log"
Oct 31 00:21:21.535475 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:21.535445    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-6hgnc_630c5695-73be-4621-b26c-523270f26c14/download-server/0.log"
Oct 31 00:21:23.431834 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:23.431788    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4l22_82ca7641-8711-4a48-86e0-a59f3ca8e498/dns/0.log"
Oct 31 00:21:23.457394 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:23.457348    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4l22_82ca7641-8711-4a48-86e0-a59f3ca8e498/kube-rbac-proxy/0.log"
Oct 31 00:21:23.567736 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:23.567703    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-czzsf_fea866f2-91be-418e-8a5f-dc609e36cfd5/dns-node-resolver/0.log"
Oct 31 00:21:24.275388 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:24.275341    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-757d6667b6-sshfm_914db174-8917-48c4-b4d2-c6cab87225a4/registry/0.log"
Oct 31 00:21:24.302186 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:24.302150    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-prlzd_6748f5ad-4980-4e4b-b448-bdc7f50e9e57/node-ca/0.log"
Oct 31 00:21:25.178328 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:25.178295    4555 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lrd6t/perf-node-gather-daemonset-qdgkm"
Oct 31 00:21:25.438678 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:25.438607    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f859hrmz_91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2/istio-proxy/0.log"
Oct 31 00:21:27.458406 ip-10-0-134-83 kubenswrapper[4555]: E1031 00:21:27.457244    4555 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wk72x" containerName="extractor"
Oct 31 00:21:28.271828 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:28.271786    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dghms_330a67ac-e5a9-4bb7-9d70-89bfd52a5efe/istio-proxy/0.log"
Oct 31 00:21:28.307944 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:28.307915    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dlhf9_cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d/istio-proxy/0.log"
Oct 31 00:21:28.463427 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:28.463398    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-mjk4b_ea819a48-154f-4c7a-a747-3306ac423289/istio-proxy/0.log"
Oct 31 00:21:28.465094 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:28.465048    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-mjk4b_ea819a48-154f-4c7a-a747-3306ac423289/istio-proxy/1.log"
Oct 31 00:21:28.615504 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:28.615436    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78_f308b264-593d-49b7-b1b9-03c09e9cee62/istio-proxy/0.log"
Oct 31 00:21:29.922125 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:29.922093    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f859hrmz_91db73c5-86c5-4ec1-a7a8-dbdd5effa7b2/istio-proxy/0.log"
Oct 31 00:21:31.136654 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:31.136605    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lgv9k_e8fa85d1-6827-43dd-a947-71d6ebc1ce6d/serve-healthcheck-canary/0.log"
Oct 31 00:21:31.887387 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:31.887352    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wk72x_5a1dd390-1d71-4270-9a6f-d9e1c66c6e52/kube-rbac-proxy/0.log"
Oct 31 00:21:31.913382 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:31.913352    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wk72x_5a1dd390-1d71-4270-9a6f-d9e1c66c6e52/exporter/0.log"
Oct 31 00:21:31.939213 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:31.939176    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wk72x_5a1dd390-1d71-4270-9a6f-d9e1c66c6e52/extractor/0.log"
Oct 31 00:21:32.303072 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:32.303044    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dghms_330a67ac-e5a9-4bb7-9d70-89bfd52a5efe/istio-proxy/0.log"
Oct 31 00:21:32.323280 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:32.323254    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dlhf9_cc70aa92-8b9e-4cf7-a66c-23b16fbcfd5d/istio-proxy/0.log"
Oct 31 00:21:32.406796 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:32.406760    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-mjk4b_ea819a48-154f-4c7a-a747-3306ac423289/istio-proxy/0.log"
Oct 31 00:21:32.408045 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:32.408024    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-mjk4b_ea819a48-154f-4c7a-a747-3306ac423289/istio-proxy/1.log"
Oct 31 00:21:32.483990 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:32.483959    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-599fb7fb95-4hx78_f308b264-593d-49b7-b1b9-03c09e9cee62/istio-proxy/0.log"
Oct 31 00:21:34.674735 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:34.674708    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6986c789f7-kp2lg_feb8e67e-7507-428f-a274-8793dee4b0d3/manager/0.log"
Oct 31 00:21:34.699925 ip-10-0-134-83 kubenswrapper[4555]: I1031 00:21:34.699891    4555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6986c789f7-kp2lg_feb8e67e-7507-428f-a274-8793dee4b0d3/copy-manifests/0.log"